Oct 30 23:38:52.990062 ip-10-0-139-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 23:38:52.990074 ip-10-0-139-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 23:38:52.990083 ip-10-0-139-66 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 23:38:52.990417 ip-10-0-139-66 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 23:39:03.188710 ip-10-0-139-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 23:39:03.188731 ip-10-0-139-66 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 98b04ec4881a4eac980a08376b815cbc --
Oct 30 23:41:13.869376 ip-10-0-139-66 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:41:14.342261 ip-10-0-139-66 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:14.342261 ip-10-0-139-66 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:41:14.342261 ip-10-0-139-66 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:14.342261 ip-10-0-139-66 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:41:14.348059 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.347925    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.348059 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348059    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.348059 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348066    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348070    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348072    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348075    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348078    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348080    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348083    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348086    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348089    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348091    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348094    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348097    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348099    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348102    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348104    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348108    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348112    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348114    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.348156 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348117    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348120    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348123    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348126    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348134    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348137    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348140    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348142    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348147    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348152    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348155    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348159    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348162    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348165    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348168    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348171    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348173    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348176    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348179    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.348606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348181    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348184    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348186    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348189    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348191    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348194    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348196    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348199    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348201    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348204    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348207    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348210    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348212    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348215    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348217    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348221    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348224    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348227    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348229    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348240    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.349067 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348243    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348246    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348249    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348252    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348255    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348257    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348260    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348263    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348265    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348267    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348271    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348275    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348278    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348281    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348285    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348288    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348290    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348294    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348297    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348299    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.349528 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348302    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348305    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348307    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348310    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348312    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348315    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348318    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348321    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348323    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348326    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348328    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348331    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348334    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348336    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348339    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348342    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348344    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348347    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348349    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348352    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.349996 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348354    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348357    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348359    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348362    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348364    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.348368    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349198    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349206    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349209    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349213    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349215    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349218    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349221    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349223    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349226    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349230    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349234    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349237    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349240    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.350530 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349243    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349246    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349249    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349251    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349254    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349257    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349259    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349262    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349265    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349267    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349270    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349272    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349275    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349277    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349280    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349282    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349284    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349287    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349289    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.350990 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349292    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349295    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349298    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349301    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349304    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349307    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349310    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349312    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349315    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349317    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349320    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349322    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349325    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349328    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349331    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349333    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349336    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349338    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349341    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349344    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.351464 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349347    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349350    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349352    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349355    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349357    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349360    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349362    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349365    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349367    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349370    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349372    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349375    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349378    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349381    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349383    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349386    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349389    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349392    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349395    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.351936 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349398    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349400    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349403    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349406    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349409    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349411    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349415    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349417    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349420    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349422    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349425    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349427    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349430    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349432    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349435    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349437    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349440    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349442    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349445    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349447    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.352417 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349450    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349453    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349455    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349458    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349460    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349465    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349468    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349470    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349473    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349476    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349478    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349481    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349484    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349487    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349492    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349979    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349984    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349988    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349992    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.352920 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349995    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.349998    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350001    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350004    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350007    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350010    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350013    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350016    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350019    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350022    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350025    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350042    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350045    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350048    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350050    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350053    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350057    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350059    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350063    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350065    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.353395 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350068    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350070    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350073    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350076    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350079    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350081    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350084    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350086    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350089    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350091    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350094    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350097    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350100    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350102    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350105    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350107    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350110    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350112    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350115    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.353862 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350118    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350120    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350123    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350125    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350128    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350131    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350134    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350138    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350141    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350145    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350148    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350152    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350155    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350158    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350162    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350166    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350169    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350171    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350174    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.354336 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350177    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350179    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350182    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350184    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350187    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350190    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350192    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350195    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350197    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350200    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350202    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350205    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350208    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350211    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350213    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350216    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350218    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350221    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350224    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.354780 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350226    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350229    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350231    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350234    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350236    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350239    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350241    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350244    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350246    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350249    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350251    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350254    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350257    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350260    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350262    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350265    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350267    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350270    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350273    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350276    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.355239 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350279    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350281    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350284    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350286    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.350289    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352129    2571 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352139    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352147    2571 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352152    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352157    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352160    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352164    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352169    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352173    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352176    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352179    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352183    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352186    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352189    2571 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352192    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352195    2571 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352198    2571 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352201    2571 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352205    2571 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:41:14.355701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352208    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352212    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352215    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352219    2571 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352223    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352226    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352237    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352241    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352244    2571 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352250    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352253    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352257    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352260    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352264    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352267    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352270    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352273    2571 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352277    2571 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352280    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352285    2571 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352288    2571 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352291    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352295    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352298    2571 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:41:14.356280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352306    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352309    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352312    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352315    2571 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352319    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352322    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352325    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352328    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352331    2571 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352335    2571 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352337    2571 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352342    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352346    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352349    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352352    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352355    2571 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352359    2571 flags.go:64] FLAG: --help="false"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352363    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352366    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352369    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352372    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352376    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352379    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:41:14.356870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352382    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352385    2571 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352388    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352391    2571 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352394    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352397    2571 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352400    2571 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352403    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352406    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352409    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352412    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352416    2571 flags.go:64] FLAG: --lock-file=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352419    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352422    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352426    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352432    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352435    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352439    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352441    2571 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352444    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352448    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352451    2571 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352454    2571 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352459    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352462    2571 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:41:14.357442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352466    2571 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352471    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352474    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352477    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352480    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352483    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352486    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352489    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352497    2571 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352500    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352503    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352506    2571 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352509    2571 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352512    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352516    2571 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352519    2571 flags.go:64] FLAG: --port="10250"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352522    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352525    2571 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-061e1a77763de1b64"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352528    2571 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352531    2571 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352534    2571 flags.go:64] FLAG: --register-node="true"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352537    2571 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352541    2571 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352544    2571 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352547    2571 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352550    2571 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:41:14.358040 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352553    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352557    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352560    2571 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352564    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352567    2571 flags.go:64] FLAG: --runonce="false"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352570    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352573    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352577    2571 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352580    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352583    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352586    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352589    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352593    2571 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352596    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352598    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352601    2571 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352604    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352607    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352610    2571 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352613    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352619    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352622    2571 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352625    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352631    2571 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352634    2571 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:41:14.358715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352637    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352640    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352643    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352646    2571 flags.go:64] FLAG: --v="2"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352650    2571 flags.go:64] FLAG: --version="false"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352655    2571 flags.go:64] FLAG: --vmodule=""
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352659    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.352662    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352822    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352827    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352830    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352838    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352842    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352844    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352848    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352851    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352854    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352856    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352859    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352861    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352864    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.359351 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352866    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352869    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352872    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352875    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352877    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352880    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352884    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352887    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352890    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352892    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352895    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352897    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352900    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352903    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352905    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352908    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352910    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352913    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352915    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.359850 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352918    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352921    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352924    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352926    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352931    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352934    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352937    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352940    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352942    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352945    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352948    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352950    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352953    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352955    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352958    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352960    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352963    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352965    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352968    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352972    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.360361 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352974    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352977    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352979    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352982    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352984    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352987    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352989    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352992    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352994    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352997    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.352999    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353002    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353005    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353007    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353010    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353012    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353015    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353018    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353021    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.360827 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353024    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353038    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353041    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353043    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353046    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353048    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353051    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353054    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353056    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353059    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353062    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353064    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353069    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353073    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353076    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353078    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353081    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353084    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353086    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353089    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.361298 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353092    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353094    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353098    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353102    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353105    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353108    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353111    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353113    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353116    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353119    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353121    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353124    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353127    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353130    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353132    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.353160    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:14.361807 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353199    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353203    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353207    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353212    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353215    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353218    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353221    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353224    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353228    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353230    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353233    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353235    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353239    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353242    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353246    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353249    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353252    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353254    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353257    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.362210 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353260    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353262    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353265    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353268    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353270    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353273    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353275    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353278    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353280    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353284    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353287    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353290    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353293    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353295    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353298    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353301    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353303    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353306    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353308    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353311    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.362661 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353313    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353317    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353319    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353322    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353325    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353327    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353330    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353333    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353335    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353339    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353342    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353344    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353347    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353349    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353352    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353354    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353357    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353360    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353362    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.363135 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353365    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353367    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353371    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353374    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353377    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353379    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353382    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353385    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353388    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353390    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353393    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353396    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353399    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353401    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353405    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353408    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353411    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353414    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353416    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353419    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.363570 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353421    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353424    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353426    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353429    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353431    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353434    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353436    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353439    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353441    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353444    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353446    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353449    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353451    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353454    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353459    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353462    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353465    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353467    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353470    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.364048 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353473    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353475    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353478    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353480    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353483    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353485    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353489    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353493    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.353496    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.360587    2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.360603    2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.360620    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.360626    2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360685    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360689    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360693    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360695    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360698    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360701    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360704    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.364511 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360706    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360709    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360712    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360715    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360717    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360720    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360723    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360726    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360729    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360732    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360735    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360738    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360740    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360743    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360746    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360749    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360751    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360754    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360757    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.365046 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360760    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360763    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360765    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360768    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360771    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360773    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360776    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360779    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360781    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360784    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360786    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360789    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360792    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360794    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360797    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360799    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360802    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360804    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360807    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.365508 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360811    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360815    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360818    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360821    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360824    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360827    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360830    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360832    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360835    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360838    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360840    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360843    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360846    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360849    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360851    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360854    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360856    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360859    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360861    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360864    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.365942 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360866    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360869    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360872    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360874    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360877    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360879    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360882    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360884    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360887    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360889    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360892    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360895    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360897    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360900    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360902    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360905    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360907    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360910    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360913    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360916    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.366550 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360918    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360921    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360923    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360926    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360928    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360931    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360933    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360936    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360938    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360941    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360944    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360946    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360949    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360952    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360954    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360958    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360962    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360965    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360968    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360970    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.367087 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.360973    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.361000    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361164    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361169    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361173    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361176    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361178    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361181    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361183    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361186    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361189    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361192    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361194    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361197    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361199    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361202    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.367606 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361205    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361207    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361210    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361212    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361215    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361217    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361221    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361223    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361226    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361229    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361231    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361233    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361236    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361240    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361244    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361246    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361249    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361252    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361254    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361257    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.368004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361259    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361262    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361264    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361267    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361269    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361272    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361274    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361277    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361280    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361284    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361286    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361289    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361291    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361294    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361297    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361300    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361303    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361305    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361311    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361314    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.368487 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361316    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361319    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361321    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361324    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361327    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361329    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361332    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361335    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361337    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361340    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361343    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361345    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361348    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361350    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361353    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361356    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361358    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361361    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361363    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.369004 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361366    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361369    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361372    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361375    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361377    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361380    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361382    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361385    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361388    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361390    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361393    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361397    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361399    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361402    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361405    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361407    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361410    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361412    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361415    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.369472 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361418    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361422    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361425    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361428    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361432    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361435    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361437    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361440    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361442    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361445    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361448    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361450    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361453    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361455    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.361478    2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361513    2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:41:14.369999 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361517    2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361521    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361525    2571 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361528    2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361530    2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361533    2571 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361536    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361538    2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361541    2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361545    2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361548    2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361550    2571 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361553    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361555    2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361558    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361561    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361563    2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361566    2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361568    2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361573    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:41:14.370683 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361576    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361579    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361581    2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361584    2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361587    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361589    2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361592    2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361594    2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361597    2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361599    2571 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361603    2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361607    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361610    2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361613    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361616    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361619    2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361621    2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361624    2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361626    2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361629    2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:41:14.371503 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361631    2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361636    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361639    2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361641    2571 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361644    2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361646    2571 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361649    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361652    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361654    2571 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361657    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361659    2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361663    2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361665    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361668    2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361671    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361673    2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361676    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361678    2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361681    2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361684    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:41:14.372014 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361686    2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361689    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361691    2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361694    2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361696    2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361699    2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361703    2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361705    2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361708    2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361711    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361713    2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361716    2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361718    2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361722    2571 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361725    2571 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361727    2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361730    2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361732    2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361735    2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:41:14.372591 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361737    2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361740    2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361742    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361745    2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361749    2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361752    2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361754    2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361757    2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361759    2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361762    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361765    2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361767    2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361770    2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361772    2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361775    2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361777    2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361780    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361784    2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361788    2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:41:14.373018 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361792    2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361795    2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361797    2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361800    2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361802    2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361805    2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:14.361807    2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.361861    2571 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.364825    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.366243    2571 server.go:1014] "Starting client certificate rotation"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.366367    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.366421    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.369182    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:41:14.373653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.373586    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:41:14.387658 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.387634    2571 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:41:14.393084 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.393066    2571 log.go:25] "Validated CRI v1 image API"
Oct 30 23:41:14.393844 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.393821    2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:41:14.396856 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.396832    2571 fs.go:135] Filesystem UUIDs: map[01cb9355-f538-4b86-8afa-e7e30c32a792:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddc21fc6-6983-4ad9-891e-3ab880f15ad2:/dev/nvme0n1p3]
Oct 30 23:41:14.396931 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.396853    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 23:41:14.401999 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.401884    2571 manager.go:217] Machine: {Timestamp:2026-10-30 23:41:14.401403658 +0000 UTC m=+0.395895130 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096562 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a59bd40b1e5d81fb8bc75ab065fdb SystemUUID:ec2a59bd-40b1-e5d8-1fb8-bc75ab065fdb BootID:98b04ec4-881a-4eac-980a-08376b815cbc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d4:14:cd:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d4:14:cd:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:84:4a:9d:98:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:41:14.402581 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.402569    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:41:14.402686 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.402660    2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:41:14.408139 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.408103    2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:41:14.408386 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.408140    2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-66.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:41:14.408434 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.408389    2571 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:41:14.408434 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.408398    2571 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:41:14.408434 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.408424    2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:41:14.409757 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.409746    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:41:14.410694 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.410684    2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:41:14.414675 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.414661    2571 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:41:14.414838 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.414828    2571 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:41:14.415662 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.415652    2571 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:41:14.415696 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.415667    2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:41:14.415696 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.415678    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:41:14.415696 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.415693    2571 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:41:14.415811 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.415702    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:41:14.416099 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.416074    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:41:14.419447 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.419428    2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:41:14.425486 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.425473    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:41:14.426401 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426378    2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:41:14.426482 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426411    2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:41:14.426580 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426569    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:41:14.426609 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426586    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:41:14.426609 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426593    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:41:14.426609 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426601    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:41:14.426691 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426610    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:41:14.426691 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.426619    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:41:14.427376 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427367    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:41:14.427411 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427377    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:41:14.427411 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427384    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:41:14.427411 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427391    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:41:14.427411 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427400    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:41:14.427411 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.427410    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:41:14.428967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.428951    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:41:14.428967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.428965    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:41:14.432828 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.432806    2571 server.go:1263] "Started kubelet"
Oct 30 23:41:14.432938 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.432882    2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:41:14.433003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.432913    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:41:14.433003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.432981    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:41:14.433654 ip-10-0-139-66 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:41:14.434295 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.434260    2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:41:14.440196 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.440166    2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 23:41:14.440702 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.440683    2571 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:41:14.444116 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.444095    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:41:14.445718 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.445703    2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:41:14.446041 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.445996    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:14.446503 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.446484    2571 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:41:14.446503 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.446504    2571 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:41:14.446640 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.446489    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:41:14.446640 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.446623    2571 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:41:14.446640 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.446633    2571 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:41:14.447217 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447201    2571 factory.go:153] Registering CRI-O factory
Oct 30 23:41:14.447217 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447217    2571 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:41:14.447347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447260    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:41:14.447347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447267    2571 factory.go:55] Registering systemd factory
Oct 30 23:41:14.447347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447273    2571 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:41:14.447347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447285    2571 factory.go:103] Registering Raw factory
Oct 30 23:41:14.447347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.447295    2571 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:41:14.448226 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.448209    2571 manager.go:319] Starting recovery of all containers
Oct 30 23:41:14.456726 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.456703    2571 manager.go:324] Recovery completed
Oct 30 23:41:14.460883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.460868    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.462796 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.462767    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.462871 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.462821    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.462871 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.462834    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.463459 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.463443    2571 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:41:14.463459 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.463456    2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:41:14.463593 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.463473    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.463593 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.463476    2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:41:14.464152 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.464137    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.464221 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.464162    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.464221 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.464171    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.465715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.465702    2571 policy_none.go:50] "Start"
Oct 30 23:41:14.465782 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.465720    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:41:14.465782 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.465738    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:41:14.467763 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.467749    2571 policy_none.go:44] "Start"
Oct 30 23:41:14.502879 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.502693    2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:41:14.506887 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.506874    2571 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:41:14.506944 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.506903    2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:41:14.506944 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.506912    2571 server.go:87] "Starting device plugin registration server"
Oct 30 23:41:14.507177 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.507165    2571 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:41:14.507215 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.507179    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:41:14.507292 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.507271    2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:41:14.507392 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.507378    2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:41:14.507392 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.507392    2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:41:14.507976 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.507956    2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:41:14.508069 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.507998    2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:14.567730 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.567684    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-66.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 23:41:14.567730 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.567674    2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-66.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 23:41:14.569810 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.567676    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.us-west-2.compute.internal.18e37323d97220e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:41:14.432774377 +0000 UTC m=+0.427265849,LastTimestamp:2026-10-30 23:41:14.432774377 +0000 UTC m=+0.427265849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.us-west-2.compute.internal,}"
Oct 30 23:41:14.582165 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.582132    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:41:14.583296 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.583268    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:41:14.583413 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.583316    2571 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:41:14.583413 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.583344    2571 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:41:14.583413 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.583388    2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 23:41:14.608251 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.608162    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.609201 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.609175    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.609306 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.609212    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.609306 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.609226    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.609306 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.609250    2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.619221 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.619190    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-66.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 23:41:14.619221 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.619144    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-66.us-west-2.compute.internal.18e37323db3c3d38  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-66.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:41:14.462797112 +0000 UTC m=+0.457288584,LastTimestamp:2026-10-30 23:41:14.462797112 +0000 UTC m=+0.457288584,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.us-west-2.compute.internal,}"
Oct 30 23:41:14.619427 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.619193    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 23:41:14.619427 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.619193    2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 23:41:14.633807 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.633772    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:41:14.633949 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.633848    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-brbkr"
Oct 30 23:41:14.684333 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.684279    2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"]
Oct 30 23:41:14.684525 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.684376    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.685469 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.685450    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.685566 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.685481    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.685566 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.685492    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.687780 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.687767    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.687945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.687926    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.687979 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.687966    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.688594 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688575    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.688594 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688589    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.688751 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688604    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.688751 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688608    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.688751 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688614    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.688751 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.688619    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.690956 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.690940    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.691011 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.690974    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:14.691792 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.691773    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:14.691867 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.691801    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:14.691867 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.691814    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:14.719999 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.719975    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.722472 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.722453    2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.732880 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.732855    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:14.738446 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.738427    2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.738515 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.738455    2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-66.us-west-2.compute.internal\": node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:14.748107 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.748081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.748176 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.748110    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.748176 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.748131    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848718 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848718 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848932 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848932 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848780    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848932 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848793    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.848932 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:14.848831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:14.855558 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.855541    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:14.955673 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:14.955597    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.023902 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.023872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:15.027056 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.027011    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:15.056576 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.056528    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.157066 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.157018    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.258170 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.258075    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.358993 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.358952    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.366140 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.366110    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 23:41:15.366322 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.366290    2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:41:15.366396 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.366300    2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ade377f6ef8974a73ad19fd7b7959d61-5b50151745721f56.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-66.us-west-2.compute.internal.18e37323db3c3d38\": read tcp 10.0.139.66:39720->13.184.210.73:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-66.us-west-2.compute.internal.18e37323db3c3d38  default   5666 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-66.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-66.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-66.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 23:41:14 +0000 UTC,LastTimestamp:2026-10-30 23:41:14.685467263 +0000 UTC m=+0.679958734,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-66.us-west-2.compute.internal,}"
Oct 30 23:41:15.460020 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.459992    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.560755 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.560699    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.639397 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.639352    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:36:15 +0000 UTC" deadline="2028-05-23 21:34:38.997818584 +0000 UTC"
Oct 30 23:41:15.639397 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.639390    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13701h53m23.358431691s"
Oct 30 23:41:15.660834 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.660808    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.733555 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.733516    2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:41:15.734653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.734636    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:41:15.734775 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.734670    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:41:15.734775 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.734683    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:41:15.760957 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.760926    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.808785 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:15.808761    2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:15.861691 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.861608    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:15.962240 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:15.962194    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:16.062558 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.062524    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:16.062743 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.062711    2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:16.163313 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.163236    2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:41:16.199619 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.199587    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e63feea4e10327df2e22d81d9a9cb4a.slice/crio-1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad WatchSource:0}: Error finding container 1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad: Status 404 returned error can't find the container with id 1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad
Oct 30 23:41:16.203423 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.203400    2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:16.204987 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.204968    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:41:16.205510 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.205487    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc66296f1ac47a99ab1c1b04d380e57.slice/crio-0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4 WatchSource:0}: Error finding container 0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4: Status 404 returned error can't find the container with id 0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4
Oct 30 23:41:16.247150 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.247119    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:16.306960 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.306922    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:16.309582 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.309295    2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:41:16.363880 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.363843    2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:41:16.417227 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.417135    2571 apiserver.go:52] "Watching apiserver"
Oct 30 23:41:16.445309 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.445277    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 23:41:16.514477 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.514442    2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:41:16.514761 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.514741    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt","openshift-multus/multus-additional-cni-plugins-lzljn","openshift-multus/network-metrics-daemon-h6w9m","openshift-network-diagnostics/network-check-target-dkcdb","openshift-ovn-kubernetes/ovnkube-node-htb47","kube-system/konnectivity-agent-w2k95","kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-gfqft","openshift-image-registry/node-ca-q6c68","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal","openshift-multus/multus-z85qc","openshift-network-operator/iptables-alerter-dp969","openshift-dns/node-resolver-lbzmk","kube-system/global-pull-secret-syncer-fjfcx"]
Oct 30 23:41:16.518564 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.518541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.520769 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.520741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.522855 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.522837    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.522956 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.522910    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:16.524888 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.524871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:16.524965 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.524929    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:16.527229 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.527211    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.529715 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.529698    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.531988 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.531971    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.534255 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.534240    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.536425 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.536406    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.538717 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.538699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.538792 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.538762    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:16.541064 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.541042    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.543244 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.543229    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.547343 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.547326    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:41:16.558133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.558230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.558230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558153    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z92tt\" (UniqueName: \"kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.558230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.558383 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.558383 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.558383 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558383 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558343    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.558383 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558371    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4bl4\" (UniqueName: \"kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558394    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4nn\" (UniqueName: \"kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558486    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g2tx\" (UniqueName: \"kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558549 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558499    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558599    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558624    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558719    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558742    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.558801 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558766    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558856    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558895    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558923    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558947    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.558990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559088    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559133 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559222    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559253    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559396    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.559489 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559501    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbsr4\" (UniqueName: \"kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559879    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.560134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.559983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560197    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfd5\" (UniqueName: \"kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p5c\" (UniqueName: \"kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560320    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2h6\" (UniqueName: \"kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.560891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560710    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.561451 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.561451 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560756    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.561451 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.560780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n999s\" (UniqueName: \"kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.568297 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.568270    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\""
Oct 30 23:41:16.568385 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.568334    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.568385 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.568278    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.569717 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.569697    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:41:16.569828 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.569776    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.569970 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.569956    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:41:16.570047 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.570016    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:41:16.570124 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.570112    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\""
Oct 30 23:41:16.570179 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.570157    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.576379 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576361    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:41:16.576450 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576430    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:41:16.576504 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576448    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.576504 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576473    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:41:16.576691 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576675    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.576817 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576804    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:41:16.576861 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.576830    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\""
Oct 30 23:41:16.578462 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.578448    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\""
Oct 30 23:41:16.578569 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.578556    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:41:16.578647 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.578636    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:41:16.580881 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.580863    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.580953 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.580899    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:41:16.581210 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.581196    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\""
Oct 30 23:41:16.581525 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.581511    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.583094 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.583082    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.583289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.583276    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.583524 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.583508    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\""
Oct 30 23:41:16.585321 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.585305    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.588803 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.588752    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerStarted","Data":"0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4"}
Oct 30 23:41:16.589800 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.589782    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"4e63feea4e10327df2e22d81d9a9cb4a","Type":"ContainerStarted","Data":"1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad"}
Oct 30 23:41:16.613439 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.613407    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:41:16.613575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.613555    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\""
Oct 30 23:41:16.613623 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.613573    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\""
Oct 30 23:41:16.614000 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.613987    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.614252 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.614233    2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:41:16.614252 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.614246    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\""
Oct 30 23:41:16.614393 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.614305    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pw64b"
Oct 30 23:41:16.661189 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.661189 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661225    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661384    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661393    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661429    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661435    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661491    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661586    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661628    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661693    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661761    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661789    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.661840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661796    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661896    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661949    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661974    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.661976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662015    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662067    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662094    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662100    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662113    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662123    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662113    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662173    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.662418 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662264    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662271    2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662120    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662230    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662214    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662424    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662401    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662414    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662551    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662396    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662592    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662599    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.663371 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662598    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662614    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662646    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662694    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662722    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662727    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbsr4\" (UniqueName: \"kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662827    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662859    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662896    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662920    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.664174 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662968    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.662984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663024    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663044    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfd5\" (UniqueName: \"kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.663070    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663086    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p5c\" (UniqueName: \"kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663123    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663146    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.663159    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.163123404 +0000 UTC m=+3.157614877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663194    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663217    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663228    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.664945 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663285    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663304    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2h6\" (UniqueName: \"kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663368    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663384    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663407    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663416    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663425    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663491    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663513    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663571    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n999s\" (UniqueName: \"kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.663605    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:16.665781 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663653    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.663698    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.163684127 +0000 UTC m=+3.158175590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663720    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663766    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663856    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92tt\" (UniqueName: \"kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663895    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663908    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663928    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.666575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663983    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.663991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bl4\" (UniqueName: \"kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664057    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664096    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4nn\" (UniqueName: \"kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664146    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664204    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664237    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g2tx\" (UniqueName: \"kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664422    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.667003 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664455    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664474    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664587    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.664593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.665341    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.665453    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.666061    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.666105    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.666164    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.667502 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.666574    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.667952 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.667937    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:41:16.668095 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.668081    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:41:16.668134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.668097    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:41:16.668167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.668151    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:41:16.673402 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.673379    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.673760 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.673745    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.722370 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.722329    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:16.722370 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.722362    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:16.722370 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.722374    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:16.722575 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:16.722476    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:17.222451026 +0000 UTC m=+3.216942502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:16.724937 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.724908    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n999s\" (UniqueName: \"kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.724937 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.724933    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbsr4\" (UniqueName: \"kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:16.725082 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.724912    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2h6\" (UniqueName: \"kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.731095 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.731065    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p5c\" (UniqueName: \"kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.731227 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.731207    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfd5\" (UniqueName: \"kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.731538 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.731518    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g2tx\" (UniqueName: \"kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.731610 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.731518    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92tt\" (UniqueName: \"kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.731610 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.731538    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4nn\" (UniqueName: \"kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.732185 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.732169    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bl4\" (UniqueName: \"kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.829651 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.829588    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:41:16.842968 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.842925    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef936d7_c04e_4aaa_a052_70ef136cbd8f.slice/crio-e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3 WatchSource:0}: Error finding container e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3: Status 404 returned error can't find the container with id e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3
Oct 30 23:41:16.853086 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.853052    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:16.859298 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.859222    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:41:16.862553 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.862523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:41:16.863319 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.863287    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790c4c63_efba_46c4_b27f_db092797bca3.slice/crio-78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a WatchSource:0}: Error finding container 78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a: Status 404 returned error can't find the container with id 78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a
Oct 30 23:41:16.869721 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.869694    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3431afb6_ad2e_44c3_a43b_2e11393e6523.slice/crio-34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77 WatchSource:0}: Error finding container 34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77: Status 404 returned error can't find the container with id 34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77
Oct 30 23:41:16.872818 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.872794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:41:16.872981 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.872931    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9019704_e296_412f_86a7_248fdcb0cfb0.slice/crio-f15713ed6c660fc7da5d4000d3cb2a14e4c6f8b6af346f06119d50726b94671c WatchSource:0}: Error finding container f15713ed6c660fc7da5d4000d3cb2a14e4c6f8b6af346f06119d50726b94671c: Status 404 returned error can't find the container with id f15713ed6c660fc7da5d4000d3cb2a14e4c6f8b6af346f06119d50726b94671c
Oct 30 23:41:16.874984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.874970    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:41:16.878933 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.878913    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z85qc"
Oct 30 23:41:16.881794 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.881776    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:41:16.883186 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.883156    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859cdafc_83b2_4a33_b5f1_b2bf87a8405c.slice/crio-c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c WatchSource:0}: Error finding container c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c: Status 404 returned error can't find the container with id c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c
Oct 30 23:41:16.883938 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.883921    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:16.885488 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.885463    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4352351_b073_4b69_a7c3_f4f46fd6cc64.slice/crio-7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef WatchSource:0}: Error finding container 7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef: Status 404 returned error can't find the container with id 7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef
Oct 30 23:41:16.890702 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.890676    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c77996_97b5_4656_9ed8_4d27ba35f8ab.slice/crio-12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f WatchSource:0}: Error finding container 12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f: Status 404 returned error can't find the container with id 12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f
Oct 30 23:41:16.894193 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.894167    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29807b1_5429_4b5e_9f35_a85384476acb.slice/crio-9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a WatchSource:0}: Error finding container 9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a: Status 404 returned error can't find the container with id 9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a
Oct 30 23:41:16.897876 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:16.897848    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb21706_15cd_4498_8e41_32d421e417e9.slice/crio-8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b WatchSource:0}: Error finding container 8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b: Status 404 returned error can't find the container with id 8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b
Oct 30 23:41:16.993746 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:16.993657    2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:41:17.168435 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.168399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:17.168435 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.168439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:17.168674 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.168552    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:17.168674 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.168554    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:17.168674 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.168607    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:18.168593254 +0000 UTC m=+4.163084716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:17.168674 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.168622    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:18.168614691 +0000 UTC m=+4.163106152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:17.269430 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.269342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:17.269606 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.269490    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:17.269606 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.269513    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:17.269606 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.269558    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:17.269766 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:17.269680    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:18.269605514 +0000 UTC m=+4.264096990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:17.593408 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.593302    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dp969" event={"ID":"c29807b1-5429-4b5e-9f35-a85384476acb","Type":"ContainerStarted","Data":"9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a"}
Oct 30 23:41:17.595304 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.595272    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z85qc" event={"ID":"26c77996-97b5-4656-9ed8-4d27ba35f8ab","Type":"ContainerStarted","Data":"12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f"}
Oct 30 23:41:17.596573 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.596545    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt" event={"ID":"d4352351-b073-4b69-a7c3-f4f46fd6cc64","Type":"ContainerStarted","Data":"7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef"}
Oct 30 23:41:17.597799 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.597772    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6c68" event={"ID":"859cdafc-83b2-4a33-b5f1-b2bf87a8405c","Type":"ContainerStarted","Data":"c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c"}
Oct 30 23:41:17.599199 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.599178    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2k95" event={"ID":"790c4c63-efba-46c4-b27f-db092797bca3","Type":"ContainerStarted","Data":"78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a"}
Oct 30 23:41:17.600593 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.600571    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b"}
Oct 30 23:41:17.601952 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.601923    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfqft" event={"ID":"b9019704-e296-412f-86a7-248fdcb0cfb0","Type":"ContainerStarted","Data":"f15713ed6c660fc7da5d4000d3cb2a14e4c6f8b6af346f06119d50726b94671c"}
Oct 30 23:41:17.603405 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.603380    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerStarted","Data":"34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77"}
Oct 30 23:41:17.604720 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.604695    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lbzmk" event={"ID":"8ef936d7-c04e-4aaa-a052-70ef136cbd8f","Type":"ContainerStarted","Data":"e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3"}
Oct 30 23:41:17.617095 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.617048    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:36:17 +0000 UTC" deadline="2028-07-06 17:23:58.161607723 +0000 UTC"
Oct 30 23:41:17.617095 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:17.617091    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14753h42m40.544522426s"
Oct 30 23:41:18.186607 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.186562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:18.186793 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.186631    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:18.186845 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.186792    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:18.186905 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.186859    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:20.186838404 +0000 UTC m=+6.181329915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:18.186961 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.186936    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:18.187013 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.186971    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:20.186958573 +0000 UTC m=+6.181450039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:18.287907 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.287860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:18.288139 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.288105    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:18.288139 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.288127    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:18.288139 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.288140    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:18.288328 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.288203    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:20.288182675 +0000 UTC m=+6.282674153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:18.603143 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.603008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:18.603646 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.603218    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:18.603646 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.603334    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:18.603646 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.603427    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:18.603646 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.603485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:18.603646 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:18.603550    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:18.621790 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.617395    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:36:17 +0000 UTC" deadline="2028-07-24 10:37:05.474505814 +0000 UTC"
Oct 30 23:41:18.621790 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:18.617464    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15178h55m46.857046727s"
Oct 30 23:41:20.205321 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.205016    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:20.205321 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.205141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:20.205321 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.205288    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:20.206059 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.205360    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.205340064 +0000 UTC m=+10.199831525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:20.206059 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.205428    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:20.206059 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.205466    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.205454875 +0000 UTC m=+10.199946335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:20.305657 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.305591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:20.305895 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.305780    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:20.305895 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.305810    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:20.305895 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.305823    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:20.305895 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.305892    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:24.305864894 +0000 UTC m=+10.300356362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:20.584616 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.584540    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:20.584805 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.584693    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:20.584805 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.584750    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:20.584933 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.584821    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:20.584933 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:20.584862    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:20.584933 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:20.584927    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:21.649591 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:21.649305    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="8b3ae473cc955cc134f7dc61f0488366941db0dc645ac8a09af3a211b59f0866" exitCode=0
Oct 30 23:41:21.650145 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:21.649693    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"8b3ae473cc955cc134f7dc61f0488366941db0dc645ac8a09af3a211b59f0866"}
Oct 30 23:41:21.663514 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:21.662472    2571 generic.go:356] "Generic (PLEG): container finished" podID="fbc66296f1ac47a99ab1c1b04d380e57" containerID="70dd65ac2e645b85f622505d8baacdb0ccfd07e4cd74af56e0fc941c28c8e61f" exitCode=0
Oct 30 23:41:21.663514 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:21.662596    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerDied","Data":"70dd65ac2e645b85f622505d8baacdb0ccfd07e4cd74af56e0fc941c28c8e61f"}
Oct 30 23:41:21.669600 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:21.669511    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"4e63feea4e10327df2e22d81d9a9cb4a","Type":"ContainerStarted","Data":"fcf68e33efccaf4680616564109321fef23f222d5454626024c9aad0c02fac93"}
Oct 30 23:41:22.584671 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.584637    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:22.584856 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:22.584758    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:22.584925 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.584904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:22.585029 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:22.585011    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:22.585127 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.585099    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:22.585205 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:22.585183    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:22.674758 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.674715    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w2k95" event={"ID":"790c4c63-efba-46c4-b27f-db092797bca3","Type":"ContainerStarted","Data":"eea48f9bf327e2e5e1929d5b78da7a682a16c2064ecb3d8a70dd834d85fd9874"}
Oct 30 23:41:22.682046 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.682008    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerStarted","Data":"585557e03a1ffe9a64617ca26c9ad5fd0bb6ed28b461d599fd8908aeb59ab3bd"}
Oct 30 23:41:22.793712 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.793591    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal" podStartSLOduration=5.793567923 podStartE2EDuration="5.793567923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:22.122033429 +0000 UTC m=+8.116524916" watchObservedRunningTime="2026-10-30 23:41:22.793567923 +0000 UTC m=+8.788059450"
Oct 30 23:41:22.913849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.912647    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" podStartSLOduration=5.912627208 podStartE2EDuration="5.912627208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:22.912255299 +0000 UTC m=+8.906746775" watchObservedRunningTime="2026-10-30 23:41:22.912627208 +0000 UTC m=+8.907118694"
Oct 30 23:41:22.913849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:22.913636    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w2k95" podStartSLOduration=2.645290387 podStartE2EDuration="7.913619205s" totalImagesPullingTime="5.268328818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:22.794995784 +0000 UTC m=+8.789487270" watchObservedRunningTime="2026-10-30 23:41:22.913619205 +0000 UTC m=+8.908110704"
Oct 30 23:41:23.689422 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:23.689388    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q6c68" event={"ID":"859cdafc-83b2-4a33-b5f1-b2bf87a8405c","Type":"ContainerStarted","Data":"c1212348ae7d989fe9f4dcb901d9999c43e94a397137a14f7a86407969002091"}
Oct 30 23:41:23.810369 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:23.809305    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q6c68" podStartSLOduration=2.33382962 podStartE2EDuration="8.80928523s" totalImagesPullingTime="6.47545561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:23.808969516 +0000 UTC m=+9.803461000" watchObservedRunningTime="2026-10-30 23:41:23.80928523 +0000 UTC m=+9.803776715"
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.242006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.242112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.242220    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.242272    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:32.242257899 +0000 UTC m=+18.236749358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.242317    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:24.242401 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.242341    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:32.242332497 +0000 UTC m=+18.236823972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:24.343293 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.342905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:24.343293 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.343152    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:24.343293 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.343172    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:24.343293 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.343186    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:24.343293 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.343248    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:32.34322798 +0000 UTC m=+18.337719446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.588798    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.588933    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.588994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.589057    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.589097    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:24.589280 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:24.589188    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:24.695465 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:24.695039    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt" event={"ID":"d4352351-b073-4b69-a7c3-f4f46fd6cc64","Type":"ContainerStarted","Data":"439606e3579031531ff4748600192a190063210e6481758ac642817a2f62b61c"}
Oct 30 23:41:26.287674 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.287616    2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:41:26.515876 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.515721    2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:41:26.287641491Z","UUID":"566e96fa-f81d-498a-acc6-55b1e3a8cc1f","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:41:26.517939 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.517902    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:41:26.518138 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.517952    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:41:26.590107 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.589759    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:26.590107 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:26.589925    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:26.590107 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.590008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:26.590107 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:26.590081    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:26.590443 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.590167    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:26.590443 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:26.590244    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:26.702571 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.702481    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z85qc" event={"ID":"26c77996-97b5-4656-9ed8-4d27ba35f8ab","Type":"ContainerStarted","Data":"89f34775aef847e53ba7471bc08b0c0cb1cfa9ffa5134619d7454893dc796e4c"}
Oct 30 23:41:26.705917 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.705880    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt" event={"ID":"d4352351-b073-4b69-a7c3-f4f46fd6cc64","Type":"ContainerStarted","Data":"9ec3c8fdc6c69ed2257e79c58c5f05daba8cb913b8f6cf1aede38617f395d4dd"}
Oct 30 23:41:26.708004 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.707934    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lbzmk" event={"ID":"8ef936d7-c04e-4aaa-a052-70ef136cbd8f","Type":"ContainerStarted","Data":"d9b484b4827b66b477eacb5693fdb9171ac5caccae00fe516abdb8650f94a8af"}
Oct 30 23:41:26.825387 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.825325    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z85qc" podStartSLOduration=2.615166889 podStartE2EDuration="11.825305927s" totalImagesPullingTime="9.210139038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:26.824945894 +0000 UTC m=+12.819437379" watchObservedRunningTime="2026-10-30 23:41:26.825305927 +0000 UTC m=+12.819797412"
Oct 30 23:41:26.948956 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:26.948854    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lbzmk" podStartSLOduration=2.372849387 podStartE2EDuration="11.948830162s" totalImagesPullingTime="9.575980775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:26.947947188 +0000 UTC m=+12.942438672" watchObservedRunningTime="2026-10-30 23:41:26.948830162 +0000 UTC m=+12.943321649"
Oct 30 23:41:27.199028 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.198945    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:27.199657 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.199619    2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:27.713169 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.713131    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="9fd57bbb054d0ab24ca4119744ced854c57ee4c930a1649261e91170726514f0" exitCode=0
Oct 30 23:41:27.714098 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.713268    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"9fd57bbb054d0ab24ca4119744ced854c57ee4c930a1649261e91170726514f0"}
Oct 30 23:41:27.714098 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.713616    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:27.714098 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:27.714091    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:41:28.587230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.587202    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:28.587230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.587219    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:28.587230 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.587213    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:28.587517 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:28.587315    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:28.587517 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:28.587392    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:28.587517 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:28.587467    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:28.716973 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.716876    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dp969" event={"ID":"c29807b1-5429-4b5e-9f35-a85384476acb","Type":"ContainerStarted","Data":"89d40de9431ca77e7a305b23a814200cd6f41ae3ff2493e591eeef660e745f5f"}
Oct 30 23:41:28.718867 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.718843    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt" event={"ID":"d4352351-b073-4b69-a7c3-f4f46fd6cc64","Type":"ContainerStarted","Data":"0c4ab9c3714de16077abb703457b7b769eac7882bc664881ead39f41ba4e5af3"}
Oct 30 23:41:28.720311 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.720288    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfqft" event={"ID":"b9019704-e296-412f-86a7-248fdcb0cfb0","Type":"ContainerStarted","Data":"967104f5d3d4c63dcc140bd505452fbbd9e4cd0bd6719bed54ecd41e103f75f3"}
Oct 30 23:41:28.769900 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.769867    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lbzmk_8ef936d7-c04e-4aaa-a052-70ef136cbd8f/dns-node-resolver/0.log"
Oct 30 23:41:28.936232 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.936179    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gfqft" podStartSLOduration=2.376090359 podStartE2EDuration="13.93616635s" totalImagesPullingTime="11.560075991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:28.936071968 +0000 UTC m=+14.930563449" watchObservedRunningTime="2026-10-30 23:41:28.93616635 +0000 UTC m=+14.930657830"
Oct 30 23:41:28.936479 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:28.936453    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dp969" podStartSLOduration=4.507970591 podStartE2EDuration="13.936446682s" totalImagesPullingTime="9.428476091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:28.825386946 +0000 UTC m=+14.819878428" watchObservedRunningTime="2026-10-30 23:41:28.936446682 +0000 UTC m=+14.930938162"
Oct 30 23:41:29.050406 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:29.048156    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt" podStartSLOduration=3.41176027 podStartE2EDuration="14.04813675s" totalImagesPullingTime="10.63637648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:29.047982348 +0000 UTC m=+15.042473830" watchObservedRunningTime="2026-10-30 23:41:29.04813675 +0000 UTC m=+15.042628232"
Oct 30 23:41:29.723881 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:29.723847    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="fae8dba104bbdc80e505562dba67395fdd36abe0b0c87ff2dbfe0de6ad446ed2" exitCode=0
Oct 30 23:41:29.724321 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:29.723949    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"fae8dba104bbdc80e505562dba67395fdd36abe0b0c87ff2dbfe0de6ad446ed2"}
Oct 30 23:41:29.787025 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:29.786995    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q6c68_859cdafc-83b2-4a33-b5f1-b2bf87a8405c/node-ca/0.log"
Oct 30 23:41:30.584856 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:30.584670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:30.585051 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:30.584699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:30.585051 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:30.584951    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:30.585051 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:30.584741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:30.585213 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:30.585066    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:30.585213 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:30.585109    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:31.731398 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:31.731353    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="545e56bbd97d7375b89550f8389cfc3264ece0b90344c22feccdfbe655b0775f" exitCode=0
Oct 30 23:41:31.731913 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:31.731400    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"545e56bbd97d7375b89550f8389cfc3264ece0b90344c22feccdfbe655b0775f"}
Oct 30 23:41:32.315944 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.315905    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:32.316139 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.316113    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:32.316139 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.316133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:32.316230 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.316191    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:41:48.316171548 +0000 UTC m=+34.310663021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:32.316271 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.316229    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:32.316311 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.316276    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:41:48.316260807 +0000 UTC m=+34.310752276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:32.416708 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.416665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:32.416898 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.416860    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:32.416898 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.416887    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:32.416898 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.416900    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:32.417047 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.416961    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:41:48.416942213 +0000 UTC m=+34.411433692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:32.584566 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.584484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:32.584566 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.584526    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:32.584774 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.584623    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:32.584774 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:32.584678    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:32.584774 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.584763    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:32.584923 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:32.584831    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:34.584784 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.584604    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:34.585271 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.584687    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:34.585271 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.584720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:34.585271 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:34.584948    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:34.585271 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:34.584847    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:34.585271 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:34.585010    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:34.740899 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.740813    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"033f6f6da2c18e25a63a9e50764735b1f70dde1dad287f1aefb49ab6f4fe165b"}
Oct 30 23:41:34.740899 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.740883    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"12506067726ccd2a442a758d7209f21dbba6c046993388afd6bc7d0808b9c3f2"}
Oct 30 23:41:34.741119 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.740918    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"ab9e4c4ea2f595835d4f9e1b76f91ce0d4e7deaba6ce94b73b6524c1687b29c5"}
Oct 30 23:41:34.741119 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.740952    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"61df695b221d7d11082bf37767cb8ae27dd2c2236079a18223b79cf57733cea2"}
Oct 30 23:41:34.741119 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.740988    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"05d13ca67ea2a6095f506bc809be1fe3806cef81aa99877d47fa3d129b2c110d"}
Oct 30 23:41:34.741119 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.741017    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"6168f08c6311aa210678c93c6536bceee785cfaf126808b376cc151c935c4189"}
Oct 30 23:41:34.743404 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.743376    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="5550bd646908414683f4ebb9f1686226202c74666c31a1a170b2e2adc21031b0" exitCode=0
Oct 30 23:41:34.743518 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:34.743446    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"5550bd646908414683f4ebb9f1686226202c74666c31a1a170b2e2adc21031b0"}
Oct 30 23:41:35.747885 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:35.747846    2571 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="199312e4f35cb914b3d2a7c8c456849e92ec184345e61a59c533308754298bc1" exitCode=0
Oct 30 23:41:35.748378 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:35.747903    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerDied","Data":"199312e4f35cb914b3d2a7c8c456849e92ec184345e61a59c533308754298bc1"}
Oct 30 23:41:36.584006 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.583982    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:36.584110 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.583983    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:36.584110 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.583994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:36.584201 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:36.584111    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:36.584234 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:36.584212    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:36.584320 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:36.584301    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:36.754143 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.754101    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"3386bee53a0504e9718b531273224c7c719707742ce9f92c2e70da40721a08ad"}
Oct 30 23:41:36.759370 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.759293    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzljn" event={"ID":"3431afb6-ad2e-44c3-a43b-2e11393e6523","Type":"ContainerStarted","Data":"74ddf844722c7b290e19df099c1c9b0c66c8adb599f328f98e414fb3db35908e"}
Oct 30 23:41:36.875387 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:36.875328    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzljn" podStartSLOduration=8.412954447 podStartE2EDuration="21.875314995s" totalImagesPullingTime="13.462360548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:36.874705911 +0000 UTC m=+22.869197415" watchObservedRunningTime="2026-10-30 23:41:36.875314995 +0000 UTC m=+22.869806493"
Oct 30 23:41:38.584173 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.584135    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:38.584173 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.584162    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:38.584850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.584151    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:38.584850 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:38.584275    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:38.584850 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:38.584341    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:38.584850 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:38.584390    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:38.768359 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.768315    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" event={"ID":"7eb21706-15cd-4498-8e41-32d421e417e9","Type":"ContainerStarted","Data":"a0b352c2de47a1871324906fb982c385541498ca4bfbbc02d920afa641da76ed"}
Oct 30 23:41:38.768612 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.768587    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:38.784653 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:38.784622    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:39.008049 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:39.007820    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-htb47" podStartSLOduration=7.0322627109999996 podStartE2EDuration="24.007804108s" totalImagesPullingTime="16.975541397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:38.960908151 +0000 UTC m=+24.955399632" watchObservedRunningTime="2026-10-30 23:41:39.007804108 +0000 UTC m=+25.002295588"
Oct 30 23:41:39.770505 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:39.770479    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:39.771021 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:39.770574    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:39.786770 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:39.786735    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:41:40.584524 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:40.584489    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:40.584693 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:40.584604    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:40.584693 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:40.584620    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:40.584693 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:40.584683    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:40.584803 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:40.584692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:40.584803 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:40.584780    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:42.584149 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:42.584115    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:42.584575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:42.584116    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:42.584575 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:42.584227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:42.584575 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:42.584211    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:42.584575 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:42.584290    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:42.584575 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:42.584370    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:44.584785 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:44.584745    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:44.585290 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:44.584933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:44.585290 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:44.584944    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:44.585290 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:44.585062    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:44.585290 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:44.585151    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:44.585290 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:44.585058    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:46.584367 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:46.584327    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:46.584367 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:46.584353    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:46.584918 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:46.584484    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:46.584918 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:46.584523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:46.584918 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:46.584603    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:46.584918 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:46.584664    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:47.398905 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.398872    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjfcx"]
Oct 30 23:41:47.399065 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.398984    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:47.399113 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:47.399084    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:47.402610 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.402581    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6w9m"]
Oct 30 23:41:47.402749 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.402667    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dkcdb"]
Oct 30 23:41:47.402749 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.402739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:47.402885 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:47.402863    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:47.402930 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:47.402910    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:47.403079 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:47.403059    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:48.316865 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:48.316835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:48.317355 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:48.316871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:48.317355 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.317018    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:48.317355 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.317046    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:48.317355 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.317112    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs podName:ca967018-daec-4727-aeea-4227b319d56c nodeName:}" failed. No retries permitted until 2026-10-30 23:42:20.317092349 +0000 UTC m=+66.311583822 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs") pod "network-metrics-daemon-h6w9m" (UID: "ca967018-daec-4727-aeea-4227b319d56c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:41:48.317355 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.317135    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret podName:681ea7db-6b38-46c2-91d2-80e27ef3c1af nodeName:}" failed. No retries permitted until 2026-10-30 23:42:20.317125297 +0000 UTC m=+66.311616758 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret") pod "global-pull-secret-syncer-fjfcx" (UID: "681ea7db-6b38-46c2-91d2-80e27ef3c1af") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:41:48.417508 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:48.417476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:48.417701 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.417669    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:41:48.417701 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.417696    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:41:48.417807 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.417709    2571 projected.go:196] Error preparing data for projected volume kube-api-access-bpwnh for pod openshift-network-diagnostics/network-check-target-dkcdb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:48.417807 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.417780    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh podName:3b9656a7-5042-408c-88a3-b607490fa46b nodeName:}" failed. No retries permitted until 2026-10-30 23:42:20.417757846 +0000 UTC m=+66.412249328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpwnh" (UniqueName: "kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh") pod "network-check-target-dkcdb" (UID: "3b9656a7-5042-408c-88a3-b607490fa46b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:41:48.584457 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:48.584383    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:48.584619 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.584484    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:48.584619 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:48.584520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:48.584619 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:48.584581    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:49.583821 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:49.583791    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:49.584249 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:49.583908    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:50.584339 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:50.584306    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:50.584737 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:50.584411    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dkcdb" podUID="3b9656a7-5042-408c-88a3-b607490fa46b"
Oct 30 23:41:50.584737 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:50.584466    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:50.584737 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:50.584549    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjfcx" podUID="681ea7db-6b38-46c2-91d2-80e27ef3c1af"
Oct 30 23:41:51.584280 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:51.584250    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:51.584431 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:41:51.584361    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6w9m" podUID="ca967018-daec-4727-aeea-4227b319d56c"
Oct 30 23:41:52.240171 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.240095    2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeReady"
Oct 30 23:41:52.240296 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.240215    2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:41:52.397506 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.397473    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9lzs"]
Oct 30 23:41:52.399877 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.399859    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8lbnc"]
Oct 30 23:41:52.400007 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.399993    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.401836 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.401817    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.445698 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.445503    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gm9ps"]
Oct 30 23:41:52.447747 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.447732    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.453707 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.453688    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:41:52.456074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.455736    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:41:52.456074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.455799    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:41:52.456074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.455967    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\""
Oct 30 23:41:52.456509 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.456489    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8lbnc"]
Oct 30 23:41:52.458714 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.458692    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:41:52.458966 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.458945    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:41:52.459978 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.459952    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9lzs"]
Oct 30 23:41:52.505168 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.505141    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gm9ps"]
Oct 30 23:41:52.543436 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543410    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.543586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543442    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.543586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.543586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.543586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543577    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.543711 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7tx\" (UniqueName: \"kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.543711 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.543711 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.543798 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543749    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.543798 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543788    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh55x\" (UniqueName: \"kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.543859 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.543832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmm5\" (UniqueName: \"kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.555346 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.555319    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:41:52.555472 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.555369    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\""
Oct 30 23:41:52.555580 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.555562    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\""
Oct 30 23:41:52.556438 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.556416    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:41:52.556546 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.556525    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:41:52.556640 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.556626    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:41:52.584437 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.584405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:41:52.584825 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.584599    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:41:52.645165 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.645353 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7tx\" (UniqueName: \"kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.645353 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645353 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645353 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.645353 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh55x\" (UniqueName: \"kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.645608 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmm5\" (UniqueName: \"kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645608 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645422    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.645608 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645608 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645780 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.645780 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645696    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.645780 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645729    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.645870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645845    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.646022 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.645997    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.646111 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.646097    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.649199 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.649178    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.649199 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.649188    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.649356 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.649309    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.655877 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.655858    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\""
Oct 30 23:41:52.655993 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.655979    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:41:52.656091 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.656079    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\""
Oct 30 23:41:52.656252 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.656240    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:41:52.656401 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.656386    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:41:52.703348 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.703322    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmm5\" (UniqueName: \"kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.703463 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.703368    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7tx\" (UniqueName: \"kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:52.703463 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.703392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh55x\" (UniqueName: \"kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.713746 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.713728    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:41:52.716238 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.716221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:41:52.759264 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:52.758831    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:53.138488 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.138411    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8lbnc"]
Oct 30 23:41:53.142118 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.142066    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gm9ps"]
Oct 30 23:41:53.142296 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:53.142273    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode530137f_bb24_4651_b3bf_b634487b3fbc.slice/crio-1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8 WatchSource:0}: Error finding container 1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8: Status 404 returned error can't find the container with id 1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8
Oct 30 23:41:53.142569 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.142549    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9lzs"]
Oct 30 23:41:53.144679 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:53.144660    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cd805f_b97f_44d4_87f2_ddb1d8ad770e.slice/crio-b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c WatchSource:0}: Error finding container b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c: Status 404 returned error can't find the container with id b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c
Oct 30 23:41:53.146023 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:53.146003    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2276a6da_3505_4ad6_a190_632d6586d916.slice/crio-d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8 WatchSource:0}: Error finding container d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8: Status 404 returned error can't find the container with id d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8
Oct 30 23:41:53.583750 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.583709    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:41:53.633701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.633665    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:41:53.633701 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.633665    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\""
Oct 30 23:41:53.801670 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.801640    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm9ps" event={"ID":"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e","Type":"ContainerStarted","Data":"b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c"}
Oct 30 23:41:53.802966 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.802946    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9lzs" event={"ID":"2276a6da-3505-4ad6-a190-632d6586d916","Type":"ContainerStarted","Data":"06cf543d779dc8099060a73088b1b407a6f7afb33079a5d3ac5702edb2ca0c8b"}
Oct 30 23:41:53.803085 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.802996    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9lzs" event={"ID":"2276a6da-3505-4ad6-a190-632d6586d916","Type":"ContainerStarted","Data":"d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8"}
Oct 30 23:41:53.803949 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:53.803930    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8lbnc" event={"ID":"e530137f-bb24-4651-b3bf-b634487b3fbc","Type":"ContainerStarted","Data":"1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8"}
Oct 30 23:41:54.316971 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.316938    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-847hn"]
Oct 30 23:41:54.319308 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.319282    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.416579 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416350    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:41:54.416579 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416350    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:41:54.416870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416539    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 23:41:54.416870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416631    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xzm7s\""
Oct 30 23:41:54.416870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416666    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 23:41:54.416870 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.416759    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:41:54.459503 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.459468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.459503 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.459507    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.459764 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.459529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.459764 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.459694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9x4\" (UniqueName: \"kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.464368 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.464252    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-847hn"]
Oct 30 23:41:54.560219 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.560187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.560376 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.560357    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9x4\" (UniqueName: \"kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.560429 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.560387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.560429 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.560406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.561168 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.561141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.563422 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.563367    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.564014 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.563988    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.619332 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.619228    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9x4\" (UniqueName: \"kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.631574 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.631545    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:41:54.808385 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:54.808350    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9lzs" event={"ID":"2276a6da-3505-4ad6-a190-632d6586d916","Type":"ContainerStarted","Data":"f602d1b8752108a1712ec82f485873b14571df2733b8b4879d6d7da75fd36e5f"}
Oct 30 23:41:55.048422 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:55.048391    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-847hn"]
Oct 30 23:41:55.052329 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:41:55.052294    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c3a0a8_b223_49c9_ad4f_0386b8ff0a9e.slice/crio-494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f WatchSource:0}: Error finding container 494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f: Status 404 returned error can't find the container with id 494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f
Oct 30 23:41:55.813891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:55.813849    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn" event={"ID":"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e","Type":"ContainerStarted","Data":"494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f"}
Oct 30 23:41:56.819243 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:56.818971    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm9ps" event={"ID":"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e","Type":"ContainerStarted","Data":"c87a5b0c805a4ae6e24f03a3bf42e9e0f8a36fc5a7efdab4f95db7d9dbc43307"}
Oct 30 23:41:56.819725 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:56.819277    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gm9ps" event={"ID":"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e","Type":"ContainerStarted","Data":"d7d15a552b21d3313d6daf978e04cc1cbc42f5258e1f2e98eddaf6d9f9300881"}
Oct 30 23:41:56.819725 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:56.819312    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:41:56.821713 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:56.821685    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9lzs" event={"ID":"2276a6da-3505-4ad6-a190-632d6586d916","Type":"ContainerStarted","Data":"f50287d3b84ea8cfe34bd33cd7447c00c1f68f215ba446580ccd94c966b0c68b"}
Oct 30 23:41:56.930500 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:56.930440    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-gm9ps" podStartSLOduration=2.257171112 podStartE2EDuration="4.930421888s" totalImagesPullingTime="2.673250776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:56.930145746 +0000 UTC m=+42.924637230" watchObservedRunningTime="2026-10-30 23:41:56.930421888 +0000 UTC m=+42.924913371"
Oct 30 23:41:57.043282 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:57.043221    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b9lzs" podStartSLOduration=1.630445943 podStartE2EDuration="5.043202427s" totalImagesPullingTime="3.412756484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:41:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:57.042399207 +0000 UTC m=+43.036890688" watchObservedRunningTime="2026-10-30 23:41:57.043202427 +0000 UTC m=+43.037693909"
Oct 30 23:41:57.825988 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:57.825952    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8lbnc" event={"ID":"e530137f-bb24-4651-b3bf-b634487b3fbc","Type":"ContainerStarted","Data":"49105c285fc173ca5d06b2a05bf0a2422dc7dd637f1dafb00777106bc0ef6f31"}
Oct 30 23:41:57.827842 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:57.827814    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn" event={"ID":"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e","Type":"ContainerStarted","Data":"6564167664b918808bad07f176e44032b89b0aa31b04019eeea93d796001627a"}
Oct 30 23:41:57.827935 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:57.827869    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn" event={"ID":"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e","Type":"ContainerStarted","Data":"ddb8c66ca64ccfb0543f42bf6641cc999ede57d2894e26005dc99cc6ad1c77eb"}
Oct 30 23:41:57.935445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:41:57.935389    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8lbnc" podStartSLOduration=1.9566326379999999 podStartE2EDuration="5.935375548s" totalImagesPullingTime="3.97874291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:57.934601935 +0000 UTC m=+43.929093426" watchObservedRunningTime="2026-10-30 23:41:57.935375548 +0000 UTC m=+43.929867028"
Oct 30 23:42:00.759977 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.759921    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn" podStartSLOduration=4.718505628 podStartE2EDuration="6.759907858s" totalImagesPullingTime="2.04140223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:58.044131357 +0000 UTC m=+44.038622839" watchObservedRunningTime="2026-10-30 23:42:00.759907858 +0000 UTC m=+46.754399339"
Oct 30 23:42:00.760490 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.760123    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"]
Oct 30 23:42:00.856928 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.856897    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"]
Oct 30 23:42:00.857100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.857010    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:00.906337 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.906299    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 23:42:00.953285 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.953258    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 23:42:00.953285 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.953258    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r2zhx\""
Oct 30 23:42:00.980614 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.980581    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"]
Oct 30 23:42:00.999766 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:00.999742    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.015303 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.015241    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.015303 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.015284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.015483 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.015308    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.015483 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.015427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrn2n\" (UniqueName: \"kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.020625 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.020601    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvtvb"]
Oct 30 23:42:01.036960 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.036936    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.057612 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.057588    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 23:42:01.057965 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.057949    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 23:42:01.058070 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.057967    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ntvcp\""
Oct 30 23:42:01.069451 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.069426    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"]
Oct 30 23:42:01.115830 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.115792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.115972 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.115837    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.116151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.116219 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.116219 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2n\" (UniqueName: \"kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.116281 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.116281 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqtw\" (UniqueName: \"kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.116375 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116294    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.116432 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.116527 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116513    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.116930 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.116903    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.120060 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.120023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.120126 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.120043    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.121331 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.121303    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4224x\""
Oct 30 23:42:01.121331 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.121327    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:42:01.121627 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.121613    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:42:01.121735 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.121719    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 23:42:01.121775 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.121741    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:42:01.174980 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.174951    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn2n\" (UniqueName: \"kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.204800 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.204757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:01.217407 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217385    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.217490 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217416    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217490 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217447    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217490 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217590 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217590 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217591    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.217659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217608    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wd9r\" (UniqueName: \"kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217768 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217714    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217768 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217756    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.217954 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.218051 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.217972    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.218051 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.218001    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.218157 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.218054    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqtw\" (UniqueName: \"kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.218157 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.218070    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.218373 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.218350    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.219268 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.218886    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.220611 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.220587    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.220730 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.220714    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.279290 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.279262    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqtw\" (UniqueName: \"kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.319322 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319322 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd9r\" (UniqueName: \"kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319451    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319550 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319585    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319586    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319655    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.319883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.319746    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.320023 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.320005    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.320089 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.320052    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.321834 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.321818    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.321891 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.321875    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.352017 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.351987    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:01.377538 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.377508    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd9r\" (UniqueName: \"kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.401586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.401550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:01.416106 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:01.416076    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc0416c_9ced_475c_8022_97cac5f04340.slice/crio-d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a WatchSource:0}: Error finding container d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a: Status 404 returned error can't find the container with id d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a
Oct 30 23:42:01.629389 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.629297    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"]
Oct 30 23:42:01.634178 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:01.634153    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cc93e9_3bb7_4224_9575_d26cf94807c7.slice/crio-a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39 WatchSource:0}: Error finding container a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39: Status 404 returned error can't find the container with id a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39
Oct 30 23:42:01.723875 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.723843    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"]
Oct 30 23:42:01.726829 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:01.726797    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cefa5e_9c3d_42fb_ad15_ad4e1b99036b.slice/crio-d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242 WatchSource:0}: Error finding container d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242: Status 404 returned error can't find the container with id d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242
Oct 30 23:42:01.842313 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.842283    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvtvb" event={"ID":"9cc0416c-9ced-475c-8022-97cac5f04340","Type":"ContainerStarted","Data":"d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a"}
Oct 30 23:42:01.843605 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.843576    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr" event={"ID":"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b","Type":"ContainerStarted","Data":"d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242"}
Oct 30 23:42:01.845665 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.845631    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv" event={"ID":"a0cc93e9-3bb7-4224-9575-d26cf94807c7","Type":"ContainerStarted","Data":"bed04faaa44ac3664aa1b0bfe2043415d6f2fd1785fc324cd485e0fae404730a"}
Oct 30 23:42:01.845764 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.845693    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv" event={"ID":"a0cc93e9-3bb7-4224-9575-d26cf94807c7","Type":"ContainerStarted","Data":"8e114d8a237afd44c1bea2a6133536fa980d5d40fdaea9932e69489e3c9c7399"}
Oct 30 23:42:01.845764 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:01.845723    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv" event={"ID":"a0cc93e9-3bb7-4224-9575-d26cf94807c7","Type":"ContainerStarted","Data":"a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39"}
Oct 30 23:42:03.202540 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.201728    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:03.236378 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.236346    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.258293 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.258265    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:03.303895 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303861    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 23:42:03.303895 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303861    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303874    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303918    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303892    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303944    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.303968    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 23:42:03.304151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.304077    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 23:42:03.304442 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.304187    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7qdml\""
Oct 30 23:42:03.336775 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336775 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336968 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336968 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336892    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336968 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336916    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336968 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336941    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.336968 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.336999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.337024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.337069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.337114    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337151 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.337140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.337294 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.337181    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.380918 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.380882    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 23:42:03.438493 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438629    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438698    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438751    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438773    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.438850 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.438827    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.439525 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.439499    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.440220 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.440193    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.441616 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.439497    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.442760    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.442851    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.443116    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.443141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443167 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.443146    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.443787 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.443764    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.444269 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.444245    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.444347 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.444308    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.448672 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.448649    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.497632 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.497564    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.553988 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.553958    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"]
Oct 30 23:42:03.572871 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.572846    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.619467 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.619439    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"]
Oct 30 23:42:03.622797 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622756    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dkdfd\""
Oct 30 23:42:03.622797 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622757    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 23:42:03.622984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622799    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 23:42:03.622984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622826    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-46rdvffkpcraf\""
Oct 30 23:42:03.622984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622835    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 23:42:03.622984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622862    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 23:42:03.622984 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.622877    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 23:42:03.654478 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.654449    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:03.741745 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.741745 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtn49\" (UniqueName: \"kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.741958 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.741958 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.741958 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.742128 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.741985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.742128 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.742073    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.742213 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.742126    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.742213 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.742163    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtn49\" (UniqueName: \"kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.843849 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.843699    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.844375 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.844169    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.844587 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.844533    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.846827 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.846777    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.846960 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.846939    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.847286 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.847264    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.847375 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.847288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.847452 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.847432    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.848922 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.848901    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.904918 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.904890    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtn49\" (UniqueName: \"kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:03.997223 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:03.997189    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:04.244881 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.244748    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:04.313303 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:04.313224    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0784b5a_8ea2_47f2_898c_f8f3240a72b2.slice/crio-dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c WatchSource:0}: Error finding container dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c: Status 404 returned error can't find the container with id dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c
Oct 30 23:42:04.405134 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.405005    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"]
Oct 30 23:42:04.410739 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:04.410713    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed2164b_7445_41c3_a449_bc2b0168ba72.slice/crio-b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7 WatchSource:0}: Error finding container b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7: Status 404 returned error can't find the container with id b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7
Oct 30 23:42:04.859743 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.859651    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv" event={"ID":"a0cc93e9-3bb7-4224-9575-d26cf94807c7","Type":"ContainerStarted","Data":"67429433b23febbaaae36faeb305d2ce7578998df2262a2fec3bebcdda3bf917"}
Oct 30 23:42:04.861664 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.861625    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7"}
Oct 30 23:42:04.863623 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.863590    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c"}
Oct 30 23:42:04.865858 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.865832    2571 generic.go:356] "Generic (PLEG): container finished" podID="9cc0416c-9ced-475c-8022-97cac5f04340" containerID="a1f9c76a494c6fc3e21cf47e9e581f1e2cf049aff222b764791ea6c8abc20f3f" exitCode=0
Oct 30 23:42:04.865971 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.865900    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvtvb" event={"ID":"9cc0416c-9ced-475c-8022-97cac5f04340","Type":"ContainerDied","Data":"a1f9c76a494c6fc3e21cf47e9e581f1e2cf049aff222b764791ea6c8abc20f3f"}
Oct 30 23:42:04.868021 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.868002    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr" event={"ID":"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b","Type":"ContainerStarted","Data":"1ef035fa9f06c9f248b01c1bd40b066d47430d492b48d9c68a2d92bb3084842d"}
Oct 30 23:42:04.868139 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.868062    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr" event={"ID":"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b","Type":"ContainerStarted","Data":"37a6246458a050763b17885f6e5baf5f65829a713d69f8cc30f29531d59a6df7"}
Oct 30 23:42:04.868139 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.868093    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr" event={"ID":"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b","Type":"ContainerStarted","Data":"8949c9cbf1d639ff24fe628e63f624ba6b8c6cbba1c256399ce9025dab537013"}
Oct 30 23:42:04.972938 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:04.972886    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv" podStartSLOduration=2.759237352 podStartE2EDuration="4.972869863s" totalImagesPullingTime="2.213632511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:04.971796168 +0000 UTC m=+50.966287649" watchObservedRunningTime="2026-10-30 23:42:04.972869863 +0000 UTC m=+50.967361343"
Oct 30 23:42:05.198829 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:05.198735    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr" podStartSLOduration=2.597164524 podStartE2EDuration="5.198719619s" totalImagesPullingTime="2.601555095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:05.197071159 +0000 UTC m=+51.191562641" watchObservedRunningTime="2026-10-30 23:42:05.198719619 +0000 UTC m=+51.193211101"
Oct 30 23:42:05.876449 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:05.876405    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvtvb" event={"ID":"9cc0416c-9ced-475c-8022-97cac5f04340","Type":"ContainerStarted","Data":"e95d0a2c261451e9c857159314e179e390d7b9151ca9d4e04a69fd68d07414b6"}
Oct 30 23:42:05.876449 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:05.876459    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvtvb" event={"ID":"9cc0416c-9ced-475c-8022-97cac5f04340","Type":"ContainerStarted","Data":"73b887af512e04fd6864c63664cafba65841a1d1e2514bf77cebc3ac378a0f8a"}
Oct 30 23:42:05.991523 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:05.991456    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvtvb" podStartSLOduration=3.533198187 podStartE2EDuration="5.991436445s" totalImagesPullingTime="2.363559403s" totalInitContainerRuntime="94.678855ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:05.990950326 +0000 UTC m=+51.985441807" watchObservedRunningTime="2026-10-30 23:42:05.991436445 +0000 UTC m=+51.985927954"
Oct 30 23:42:06.252961 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.252926    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mxk2r"]
Oct 30 23:42:06.265761 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.265737    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.314730 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.314689    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mxk2r"]
Oct 30 23:42:06.315635 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.315178    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-mw499\""
Oct 30 23:42:06.315635 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.315240    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:42:06.315635 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.315354    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:42:06.366329 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.366289    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h7m\" (UniqueName: \"kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.366515 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.366349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.467236 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.467206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h7m\" (UniqueName: \"kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.467367 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.467261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.467641 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.467621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.531137 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.531103    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h7m\" (UniqueName: \"kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.595602 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.595566    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:06.667106 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.666448    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54885d6898-mpk4r"]
Oct 30 23:42:06.685301 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.685274    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.730317 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.730283    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54885d6898-mpk4r"]
Oct 30 23:42:06.735074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.734994    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 23:42:06.735074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.735006    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 23:42:06.735074 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.735064    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 23:42:06.735355 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.735111    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 23:42:06.735355 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.735209    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c266j\""
Oct 30 23:42:06.735355 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.735265    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 23:42:06.769955 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.769910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.769955 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.769954    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qnn\" (UniqueName: \"kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770261 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.769985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770261 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.770170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770261 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.770208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770261 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.770240    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770445 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.770387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.770523 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.770440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.782620 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.782586    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 23:42:06.830581 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.830547    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:06.871568 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871530    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.871816 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.871886 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.871886 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.871982 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.871982 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.871978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.872105 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.872075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.872149 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.872099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qnn\" (UniqueName: \"kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.872600 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.872575    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.872939 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.872914    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.872939 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.872929    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.874966 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.874938    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.875126 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.875105    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.875370 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.875348    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.875471 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.875385    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:06.879852 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.879823    2571 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870" exitCode=0
Oct 30 23:42:06.880196 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.879945    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870"}
Oct 30 23:42:06.931316 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:06.931283    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qnn\" (UniqueName: \"kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:07.008132 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.008089    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:07.013598 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.013566    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mxk2r"]
Oct 30 23:42:07.018443 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:07.018407    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfeeee0_f0b9_4317_8ada_11b67a839acf.slice/crio-ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a WatchSource:0}: Error finding container ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a: Status 404 returned error can't find the container with id ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a
Oct 30 23:42:07.447927 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.447854    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54885d6898-mpk4r"]
Oct 30 23:42:07.492128 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:07.492096    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389e6e7e_8415_4d41_8583_3bae15af7e96.slice/crio-b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e WatchSource:0}: Error finding container b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e: Status 404 returned error can't find the container with id b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e
Oct 30 23:42:07.884539 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.884501    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r" event={"ID":"389e6e7e-8415-4d41-8583-3bae15af7e96","Type":"ContainerStarted","Data":"b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e"}
Oct 30 23:42:07.885799 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.885774    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mxk2r" event={"ID":"adfeeee0-f0b9-4317-8ada-11b67a839acf","Type":"ContainerStarted","Data":"ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a"}
Oct 30 23:42:07.887822 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.887801    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"6e9026fd7cb3f603b8386396330822ecfdc547623904002232098f5ca3854faf"}
Oct 30 23:42:07.887909 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.887845    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"15de20ad9cd7d1be258c87c9e25ffeb38880791056c551a41964c4017d257107"}
Oct 30 23:42:07.887909 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:07.887870    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"cec3599aafe78e7bc4c159add68eebfb720415f63a7b1bafa138ded9a9c6ef4c"}
Oct 30 23:42:09.901618 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.901529    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"66b46503354770d3198067b315f8a93967422e3702822fe18f77e81b374c8b49"}
Oct 30 23:42:09.902165 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.901647    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"37da576b0485275e535d1a347d1240afb263c4bb7cfcb2ed8cab3485d83dec15"}
Oct 30 23:42:09.902165 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.901692    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" event={"ID":"7ed2164b-7445-41c3-a449-bc2b0168ba72","Type":"ContainerStarted","Data":"fc2103f25a0146a60cc50753551208c2bc6f0526803afe7856a0858d6e201dba"}
Oct 30 23:42:09.902165 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.901834    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:09.906459 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906428    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"}
Oct 30 23:42:09.906559 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906503    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"}
Oct 30 23:42:09.906559 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906532    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"}
Oct 30 23:42:09.906659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906559    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"}
Oct 30 23:42:09.906659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906593    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"}
Oct 30 23:42:09.906659 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:09.906616    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerStarted","Data":"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"}
Oct 30 23:42:10.033717 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.033658    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6" podStartSLOduration=2.470723415 podStartE2EDuration="7.033639168s" totalImagesPullingTime="4.562915753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:10.032051852 +0000 UTC m=+56.026543336" watchObservedRunningTime="2026-10-30 23:42:10.033639168 +0000 UTC m=+56.028130652"
Oct 30 23:42:10.083961 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.083923    2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:42:10.087539 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.087510    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115565 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115565 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115773 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115773 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115617    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115773 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115651    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115883 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115965 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115910    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115965 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.115965 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.115980    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116172 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116268 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116333 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116282    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116386 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116352    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116431 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116476 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116445    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.116533 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.116514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.138124 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.138089    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 23:42:10.138319 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.138250    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 23:42:10.138398 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.138329    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 23:42:10.138470 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.138453    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 23:42:10.186911 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.186835    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 23:42:10.186911 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.186892    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 23:42:10.186911 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.186835    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 23:42:10.186911 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.186834    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 23:42:10.187245 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.186919    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 23:42:10.217727 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217727 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217741    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217771    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217812    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.217967 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.217987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218143    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.218289 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.223695 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.218248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.223695 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.220456    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.225100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.224150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.225100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.224205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.225100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.224245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.225100 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.224282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.230095 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.229288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.231237 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.231133    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.231352 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.231324    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.231616 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.231589    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.231708 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.231647    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.231952 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.231926    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 23:42:10.232746 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.232283    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.233590 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.233081    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 23:42:10.234318 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.234294    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 23:42:10.234840 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.234817    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.236670 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.235090    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qbwc4\""
Oct 30 23:42:10.236787 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.235365    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rm397jpo960s\""
Oct 30 23:42:10.236861 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.235378    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 23:42:10.236861 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.235446    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 23:42:10.236861 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.235865    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 23:42:10.238639 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.238594    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.241107 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.240950    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:42:10.241433 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.241412    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.242636 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.241894    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.682538177 podStartE2EDuration="7.241877728s" totalImagesPullingTime="4.544755381s" totalInitContainerRuntime="14.58417ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:10.239298427 +0000 UTC m=+56.233789909" watchObservedRunningTime="2026-10-30 23:42:10.241877728 +0000 UTC m=+56.236369210"
Oct 30 23:42:10.242636 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.242442    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.242636 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.242616    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.244496 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.244477    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.245313 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.245278    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.246152 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.245794    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.246152 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.246110    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.247852 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.247828    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.248776 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.248754    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.249041 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.249013    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.282480 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.282450    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.419965 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.419933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:10.867265 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.867180    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:42:10.869574 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:10.869543    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5200d5bb_5437_456e_93bb_3c95d0e19d1d.slice/crio-8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f WatchSource:0}: Error finding container 8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f: Status 404 returned error can't find the container with id 8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f
Oct 30 23:42:10.913586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.912977    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r" event={"ID":"389e6e7e-8415-4d41-8583-3bae15af7e96","Type":"ContainerStarted","Data":"1c9c6978a06b61ce06870737bdbab8fb2365c64048e03e44a54118b4dbd69465"}
Oct 30 23:42:10.913586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.913085    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r" event={"ID":"389e6e7e-8415-4d41-8583-3bae15af7e96","Type":"ContainerStarted","Data":"02b0eecc64075bed674cc99f8029c6c0717d01ece7e40ccacc140ff5153ed4fe"}
Oct 30 23:42:10.913586 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.913122    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r" event={"ID":"389e6e7e-8415-4d41-8583-3bae15af7e96","Type":"ContainerStarted","Data":"daed0d3f818699a0806f471649834b284409aae55a545c53bb672524ea752e4a"}
Oct 30 23:42:10.916925 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:10.916895    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f"}
Oct 30 23:42:11.032590 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:11.032537    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r" podStartSLOduration=2.265179575 podStartE2EDuration="5.032520658s" totalImagesPullingTime="2.767341083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:11.030912063 +0000 UTC m=+57.025403543" watchObservedRunningTime="2026-10-30 23:42:11.032520658 +0000 UTC m=+57.027012130"
Oct 30 23:42:11.786788 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:11.786759    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:11.935222 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:11.935180    2571 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd" exitCode=0
Oct 30 23:42:11.935804 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:11.935356    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd"}
Oct 30 23:42:15.924674 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:15.924641    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:16.962244 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962204    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"}
Oct 30 23:42:16.962721 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962283    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"}
Oct 30 23:42:16.962721 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962326    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"}
Oct 30 23:42:16.962721 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962369    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"}
Oct 30 23:42:16.962721 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962411    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"}
Oct 30 23:42:16.962721 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:16.962448    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerStarted","Data":"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"}
Oct 30 23:42:17.089380 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:17.089321    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.070605897 podStartE2EDuration="8.089301959s" totalImagesPullingTime="3.997940137s" totalInitContainerRuntime="20.755925ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:17.087324454 +0000 UTC m=+63.081815947" watchObservedRunningTime="2026-10-30 23:42:17.089301959 +0000 UTC m=+63.083793443"
Oct 30 23:42:20.333396 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.333353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:20.333827 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.333468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:20.420232 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.420190    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:20.429108 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.429073    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:42:20.429108 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.429096    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:42:20.434657 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.434621    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:20.436804 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.436776    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:20.436920 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.436776    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:20.483441 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.483406    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:42:20.534045 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.534000    2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:42:20.538806 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.538774    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:20.544603 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.544282    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\""
Oct 30 23:42:20.546375 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.546355    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:20.644619 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.644593    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\""
Oct 30 23:42:20.646712 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.646692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:20.849217 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.849184    2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\""
Oct 30 23:42:20.851286 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.851265    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:20.923150 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.923118    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjfcx"]
Oct 30 23:42:20.933682 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:20.933654    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681ea7db_6b38_46c2_91d2_80e27ef3c1af.slice/crio-cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2 WatchSource:0}: Error finding container cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2: Status 404 returned error can't find the container with id cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2
Oct 30 23:42:20.980770 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:20.980733    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fjfcx" event={"ID":"681ea7db-6b38-46c2-91d2-80e27ef3c1af","Type":"ContainerStarted","Data":"cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2"}
Oct 30 23:42:21.027901 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:21.027866    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6w9m"]
Oct 30 23:42:21.031546 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:21.031515    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca967018_daec_4727_aeea_4227b319d56c.slice/crio-6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf WatchSource:0}: Error finding container 6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf: Status 404 returned error can't find the container with id 6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf
Oct 30 23:42:21.223712 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:21.223679    2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dkcdb"]
Oct 30 23:42:21.226556 ip-10-0-139-66 kubenswrapper[2571]: W1030 23:42:21.226524    2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9656a7_5042_408c_88a3_b607490fa46b.slice/crio-55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e WatchSource:0}: Error finding container 55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e: Status 404 returned error can't find the container with id 55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e
Oct 30 23:42:21.987145 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:21.987104    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dkcdb" event={"ID":"3b9656a7-5042-408c-88a3-b607490fa46b","Type":"ContainerStarted","Data":"55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e"}
Oct 30 23:42:21.988416 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:21.988387    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6w9m" event={"ID":"ca967018-daec-4727-aeea-4227b319d56c","Type":"ContainerStarted","Data":"6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf"}
Oct 30 23:42:22.752555 ip-10-0-139-66 kubenswrapper[2571]: E1030 23:42:22.752443    2571 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:42:24.000186 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:24.000144    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6w9m" event={"ID":"ca967018-daec-4727-aeea-4227b319d56c","Type":"ContainerStarted","Data":"701f482aeec15867a59d57ca86b6db8d3b7ed1fb9d4f1952546c29f6730b317a"}
Oct 30 23:42:24.000654 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:24.000209    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6w9m" event={"ID":"ca967018-daec-4727-aeea-4227b319d56c","Type":"ContainerStarted","Data":"f7cf0004b3b0d7dad59e8892037046292c436a5b6312ff83a123374f86bd3afe"}
Oct 30 23:42:24.112612 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:24.112546    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h6w9m" podStartSLOduration=67.272012199 podStartE2EDuration="1m9.112522399s" totalImagesPullingTime="1.8405102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:24.110840313 +0000 UTC m=+70.105331796" watchObservedRunningTime="2026-10-30 23:42:24.112522399 +0000 UTC m=+70.107013881"
Oct 30 23:42:27.016570 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.016517    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mxk2r" event={"ID":"adfeeee0-f0b9-4317-8ada-11b67a839acf","Type":"ContainerStarted","Data":"fc40f5f12db046110c92cb349e083cc03f959e8511b628746d36c7fd9ef802be"}
Oct 30 23:42:27.017047 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.016899    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:27.019422 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.019390    2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dkcdb" event={"ID":"3b9656a7-5042-408c-88a3-b607490fa46b","Type":"ContainerStarted","Data":"d4d6e60b5e9f0ed330e700d75f5aa86629419299579ba34c67247539bc5891ed"}
Oct 30 23:42:27.019567 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.019533    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:27.034471 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.034441    2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:27.128661 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.128604    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mxk2r" podStartSLOduration=1.656140722 podStartE2EDuration="21.128587623s" totalImagesPullingTime="19.472446901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.127308969 +0000 UTC m=+73.121800454" watchObservedRunningTime="2026-10-30 23:42:27.128587623 +0000 UTC m=+73.123079105"
Oct 30 23:42:27.238172 ip-10-0-139-66 kubenswrapper[2571]: I1030 23:42:27.238120    2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dkcdb" podStartSLOduration=67.398379994 podStartE2EDuration="1m12.238102327s" totalImagesPullingTime="4.839722333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:27.237152246 +0000 UTC m=+73.231643726" watchObservedRunningTime="2026-10-30 23:42:27.238102327 +0000 UTC m=+73.232593809"
Oct 30 23:42:27.575629 ip-10-0-139-66 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 23:42:27.589634 ip-10-0-139-66 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 23:42:27.589839 ip-10-0-139-66 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 23:42:27.590077 ip-10-0-139-66 systemd[1]: kubelet.service: Consumed 4.810s CPU time.
Oct 30 23:42:27.610669 ip-10-0-139-66 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:42:27.707016 ip-10-0-139-66 kubenswrapper[6033]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:42:27.707016 ip-10-0-139-66 kubenswrapper[6033]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:42:27.707016 ip-10-0-139-66 kubenswrapper[6033]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:42:27.707016 ip-10-0-139-66 kubenswrapper[6033]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:42:27.709829 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709724    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.709829 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709826    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.709829 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709833    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.709829 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709837    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709841    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709844    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709847    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709850    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709853    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709856    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709859    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709861    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709864    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709867    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709870    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709873    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709875    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709878    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709880    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709885    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709889    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709893    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.710190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709898    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709903    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709908    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709912    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709916    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709918    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709931    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709937    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709941    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709946    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709951    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709956    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709960    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709965    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709970    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709975    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709979    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709984    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709989    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.710879 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709993    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.709997    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710005    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710012    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710017    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710022    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710040    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710046    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710050    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710053    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710056    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710059    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710062    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710065    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710068    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710070    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710073    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710076    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710078    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.711554 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710081    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710085    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710088    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710091    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710099    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710103    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710107    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710114    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710119    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710124    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710127    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710130    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710133    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710136    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710140    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710146    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710151    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710156    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710161    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.712278 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710165    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710170    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710175    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710179    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710183    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710188    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710192    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710197    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710201    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710204    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710207    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710210    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710213    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710216    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710220    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710225    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710230    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710236    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710240    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710244    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.713064 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710247    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710251    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710256    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710260    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710265    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710269    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.710274    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711325    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711334    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711337    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711340    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711343    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711345    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711348    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711351    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711354    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711357    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711362    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711366    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711370    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.713975 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711374    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711378    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711385    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711391    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711396    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711400    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711405    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711408    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711412    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711418    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711421    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711424    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711429    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711433    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711438    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711443    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711447    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711452    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711457    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.714841 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711460    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711463    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711466    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711468    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711471    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711473    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711476    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711479    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711483    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711487    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711491    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711495    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711499    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711504    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711507    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711510    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711514    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711519    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711523    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.715662 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711527    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711531    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711536    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711544    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711549    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711554    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711559    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711564    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711570    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711575    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711580    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711585    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711589    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711595    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711602    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711607    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711612    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711616    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711621    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.716440 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711625    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711629    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711633    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711637    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711641    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711645    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711650    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711654    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711659    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711664    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711668    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711672    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711676    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711680    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711684    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711689    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711695    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711700    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711704    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711710    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.717114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711714    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711719    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711722    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711725    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711728    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711731    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711736    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711739    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711742    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711746    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711750    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711754    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711759    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711762    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711765    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.711767    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712957    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712972    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712979    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712984    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.717963 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712989    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.712994    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713000    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713012    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713016    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713021    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713040    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713045    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713050    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713055    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713060    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713066    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713070    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713074    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713077    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713080    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713086    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713089    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713092    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.718692 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713094    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713097    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713100    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713104    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713109    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713113    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713118    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713122    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713127    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713132    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713142    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713146    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713150    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713155    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713159    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713164    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713171    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713178    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713183    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.719384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713188    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713192    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713195    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713204    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713210    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713215    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713221    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713226    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713230    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713235    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713240    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713249    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713255    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713259    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713264    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713274    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713278    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713284    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713288    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.720051 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713293    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713298    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713303    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713307    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713311    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713319    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713324    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713329    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713333    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713342    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713347    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713353    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713357    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713363    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713367    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713372    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713376    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713381    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713387    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713392    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713399    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.720686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713403    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713413    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713418    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713422    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713427    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713432    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713437    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713442    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713446    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713451    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713462    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713468    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713473    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713483    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713487    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713492    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713501    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713507    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713512    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713516    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.721501 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713521    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713526    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713530    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.713535    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713743    6033 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713764    6033 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713774    6033 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713783    6033 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713790    6033 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713796    6033 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713806    6033 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713817    6033 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713823    6033 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713834    6033 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713841    6033 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713846    6033 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713851    6033 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713857    6033 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713863    6033 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713869    6033 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713874    6033 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713880    6033 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713893    6033 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:42:27.722249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713900    6033 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713906    6033 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713911    6033 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713917    6033 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713924    6033 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713932    6033 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713939    6033 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713949    6033 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713954    6033 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713959    6033 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713964    6033 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713970    6033 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713976    6033 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713984    6033 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713989    6033 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.713999    6033 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714005    6033 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714010    6033 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714015    6033 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714023    6033 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714045    6033 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714051    6033 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714055    6033 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714059    6033 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:42:27.723119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714066    6033 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714069    6033 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714073    6033 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714076    6033 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714080    6033 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714083    6033 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714086    6033 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714089    6033 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714095    6033 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714098    6033 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714103    6033 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714111    6033 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714117    6033 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714123    6033 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714126    6033 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714130    6033 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714133    6033 flags.go:64] FLAG: --help="false"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714139    6033 flags.go:64] FLAG: --hostname-override="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714142    6033 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714146    6033 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714149    6033 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714154    6033 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714160    6033 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:42:27.724109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714166    6033 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714169    6033 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714175    6033 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714178    6033 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714182    6033 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714191    6033 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714197    6033 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714203    6033 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714208    6033 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714214    6033 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714219    6033 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714228    6033 flags.go:64] FLAG: --lock-file=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714234    6033 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714239    6033 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714244    6033 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714254    6033 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714259    6033 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714265    6033 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714271    6033 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714277    6033 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714280    6033 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714285    6033 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714292    6033 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714300    6033 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714306    6033 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:42:27.724973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714312    6033 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714317    6033 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714323    6033 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714333    6033 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714336    6033 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714340    6033 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714343    6033 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714346    6033 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714360    6033 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714365    6033 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714376    6033 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714382    6033 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714389    6033 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714394    6033 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714400    6033 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714406    6033 flags.go:64] FLAG: --port="10250"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714412    6033 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714417    6033 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-061e1a77763de1b64"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714427    6033 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714433    6033 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714441    6033 flags.go:64] FLAG: --register-node="true"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714447    6033 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714455    6033 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714460    6033 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714466    6033 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714471    6033 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:42:27.725809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714482    6033 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714488    6033 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714493    6033 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714501    6033 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714507    6033 flags.go:64] FLAG: --runonce="false"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714512    6033 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714518    6033 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714524    6033 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714529    6033 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714540    6033 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714546    6033 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714551    6033 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714556    6033 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714562    6033 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714567    6033 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714572    6033 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714577    6033 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714581    6033 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714589    6033 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714592    6033 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714602    6033 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714608    6033 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714613    6033 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714622    6033 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714631    6033 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:42:27.726684 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714637    6033 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714642    6033 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714648    6033 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714655    6033 flags.go:64] FLAG: --v="2"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714663    6033 flags.go:64] FLAG: --version="false"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714670    6033 flags.go:64] FLAG: --vmodule=""
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714677    6033 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.714687    6033 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715308    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715321    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715329    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715333    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715337    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715341    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715346    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715350    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715354    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715359    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715363    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715367    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715371    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.727636 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715375    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715380    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715384    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715389    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715396    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715400    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715405    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715409    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715413    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715418    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715421    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715424    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715426    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715429    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715432    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715434    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715437    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715439    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715442    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.728403 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715445    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715447    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715452    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715455    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715457    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715460    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715463    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715466    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715469    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715472    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715475    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715478    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715480    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715483    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715487    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715491    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715495    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715501    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715504    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.729104 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715506    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715509    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715513    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715518    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715522    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715526    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715530    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715534    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715539    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715543    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715551    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715556    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715560    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715563    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715566    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715572    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715577    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715581    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715585    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.729778 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715589    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715594    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715598    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715603    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715607    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715611    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715615    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715619    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715625    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715630    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715635    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715640    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715644    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715648    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715652    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715656    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715660    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715664    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715669    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.730445 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715675    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715680    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715687    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715691    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715694    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715698    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715703    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715707    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715712    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715720    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715724    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715728    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715731    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715736    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715740    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715744    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.731190 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715748    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.715800    6033 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715859    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715869    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715874    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715879    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715883    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715887    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715893    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715897    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715901    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715904    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715910    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715914    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715919    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715923    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.731801 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715927    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715932    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715936    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715942    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715947    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715953    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715959    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715964    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715969    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715973    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715979    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715984    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715988    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715993    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.715998    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716003    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716007    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716012    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716017    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716022    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.732412 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716049    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716055    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716059    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716064    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716070    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716074    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716078    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716083    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716087    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716090    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716095    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716099    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716104    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716108    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716112    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716117    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716122    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716127    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716132    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716136    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.733106 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716141    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716147    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716154    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716159    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716162    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716165    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716168    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716170    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716173    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716176    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716178    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716181    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716184    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716186    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716189    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716191    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716195    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716198    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716201    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716204    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.733851 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716206    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716209    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716214    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716218    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716223    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716227    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716230    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716233    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716236    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716239    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716241    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716244    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716246    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716249    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716254    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716258    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716263    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716267    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716271    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.734534 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716275    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716280    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716284    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716288    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716293    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716302    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716309    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716315    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716319    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716326    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716330    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716335    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.716338    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.721218    6033 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.721243    6033 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.721268    6033 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.721274    6033 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721353    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721361    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.735236 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721366    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721371    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721376    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721380    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721384    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721389    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721393    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721399    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721403    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721407    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721410    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721413    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721416    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721418    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721421    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721424    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721427    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721429    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721432    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721436    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.736112 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721438    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721441    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721444    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721446    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721449    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721451    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721454    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721466    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721471    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721476    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721480    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721484    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721488    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721493    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721500    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721508    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721516    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721521    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721525    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.736930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721528    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721531    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721534    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721537    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721540    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721542    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721545    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721551    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721556    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721561    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721565    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721570    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721574    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721577    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721580    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721583    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721585    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721588    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721591    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721595    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.737707 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721597    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721601    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721605    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721609    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721614    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721618    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721622    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721625    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721627    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721630    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721633    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721636    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721639    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721642    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721646    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721650    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721655    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721659    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721662    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721665    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.738467 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721668    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721671    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721674    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721676    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721679    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721682    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721684    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721687    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721691    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721695    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721699    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721703    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721707    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721711    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721717    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721722    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721725    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721728    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721730    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.739205 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721733    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721736    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721739    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721741    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721744    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721747    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.721784    6033 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721973    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721981    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721986    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721990    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.721997    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722002    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722005    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722009    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722013    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.739895 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722018    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722022    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722041    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722046    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722052    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722056    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722061    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722066    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722070    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722074    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722078    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722081    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722086    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722089    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722092    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722094    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722097    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722100    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722103    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.740446 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722107    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722111    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722115    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722118    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722121    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722124    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722126    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722129    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722132    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722135    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722137    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722140    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722143    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722146    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722149    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722151    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722154    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722157    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722160    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722162    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.741156 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722165    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722167    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722170    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722173    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722175    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722178    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722181    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722185    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722189    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722194    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722200    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722206    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722211    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722216    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722220    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722225    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722230    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722235    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722240    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.741865 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722244    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722249    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722255    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722260    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722265    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722270    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722275    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722279    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722282    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722287    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722291    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722294    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722297    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722300    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722303    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722305    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722308    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722311    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722313    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722316    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.742664 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722320    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722323    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722326    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722329    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722331    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722334    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722337    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722339    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722342    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722345    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722348    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722351    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722353    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722356    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722361    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722364    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722366    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722369    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.743434 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722372    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.722418    6033 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722460    6033 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722465    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722469    6033 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722472    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722474    6033 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722477    6033 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722480    6033 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722483    6033 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722485    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722488    6033 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722492    6033 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722494    6033 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722497    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722501    6033 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:42:27.744142 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722504    6033 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722507    6033 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722510    6033 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722513    6033 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722517    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722520    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722525    6033 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722529    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722533    6033 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722538    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722541    6033 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722545    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722552    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722556    6033 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722561    6033 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722565    6033 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722570    6033 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722574    6033 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722578    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:42:27.744725 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722582    6033 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722587    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722592    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722596    6033 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722601    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722606    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722610    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722613    6033 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722616    6033 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722619    6033 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722621    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722624    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722628    6033 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722632    6033 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722636    6033 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722641    6033 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722645    6033 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722650    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722654    6033 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722658    6033 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:42:27.745348 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722663    6033 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722667    6033 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722672    6033 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722676    6033 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722679    6033 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722683    6033 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722686    6033 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722689    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722691    6033 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722694    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722697    6033 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722700    6033 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722702    6033 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722705    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722709    6033 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722713    6033 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722717    6033 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722722    6033 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722726    6033 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:42:27.746092 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722729    6033 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722732    6033 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722736    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722740    6033 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722745    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722751    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722755    6033 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722759    6033 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722763    6033 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722767    6033 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722771    6033 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722776    6033 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722780    6033 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722784    6033 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722789    6033 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722793    6033 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722797    6033 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722802    6033 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722809    6033 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:42:27.746789 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722814    6033 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722818    6033 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722825    6033 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722831    6033 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722837    6033 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722842    6033 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722847    6033 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722851    6033 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722855    6033 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722860    6033 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722865    6033 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722870    6033 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722874    6033 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722879    6033 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:27.722884    6033 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.722971    6033 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724278    6033 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724356    6033 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 23:42:27.747582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724601    6033 server.go:1014] "Starting client certificate rotation"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724768    6033 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724821    6033 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:36:15 +0000 UTC" deadline="2028-05-04 11:58:11.029249754 +0000 UTC"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.724859    6033 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13236h15m43.304393121s"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.725177    6033 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.726756    6033 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.728415    6033 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.730848    6033 log.go:25] "Validated CRI v1 image API"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.732057    6033 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:42:27.748397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.736087    6033 fs.go:135] Filesystem UUIDs: map[01cb9355-f538-4b86-8afa-e7e30c32a792:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ddc21fc6-6983-4ad9-891e-3ab880f15ad2:/dev/nvme0n1p3]
Oct 30 23:42:27.748829 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.736120    6033 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e/userdata/shm major:0 minor:502 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf/userdata/shm major:0 minor:500 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f/userdata/shm major:0 minor:452 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2/userdata/shm major:0 minor:498 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~projected/kube-api-access-sxmm5:{mountpoint:/var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~projected/kube-api-access-sxmm5 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26c77996-97b5-4656-9ed8-4d27ba35f8ab/volumes/kubernetes.io~projected/kube-api-access-n999s:{mountpoint:/var/lib/kubelet/pods/26c77996-97b5-4656-9ed8-4d27ba35f8ab/volumes/kubernetes.io~projected/kube-api-access-n999s major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3431afb6-ad2e-44c3-a43b-2e11393e6523/volumes/kubernetes.io~projected/kube-api-access-v4bl4:{mountpoint:/var/lib/kubelet/pods/3431afb6-ad2e-44c3-a43b-2e11393e6523/volumes/kubernetes.io~projected/kube-api-access-v4bl4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~projected/kube-api-access-p9qnn:{mountpoint:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~projected/kube-api-access-p9qnn major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b9656a7-5042-408c-88a3-b607490fa46b/volumes/kubernetes.io~projected/kube-api-access-bpwnh:{mountpoint:/var/lib/kubelet/pods/3b9656a7-5042-408c-88a3-b607490fa46b/volumes/kubernetes.io~projected/kube-api-access-bpwnh major:0 minor:497 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/config-reloader/2 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/prometheus/15 major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/kube-api-access-lz8wn:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/kube-api-access-lz8wn major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/tls-assets major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/config major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/web-config major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/681ea7db-6b38-46c2-91d2-80e27ef3c1af/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/681ea7db-6b38-46c2-91d2-80e27ef3c1af/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/790c4c63-efba-46c4-b27f-db092797bca3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/790c4c63-efba-46c4-b27f-db092797bca3/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~projected/kube-api-access-s4p5c:{mountpoint:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~projected/kube-api-access-s4p5c major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~projected/kube-api-access-rtn49:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~projected/kube-api-access-rtn49 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/859cdafc-83b2-4a33-b5f1-b2bf87a8405c/volumes/kubernetes.io~projected/kube-api-access-gr2h6:{mountpoint:/var/lib/kubelet/pods/859cdafc-83b2-4a33-b5f1-b2bf87a8405c/volumes/kubernetes.io~projected/kube-api-access-gr2h6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ef936d7-c04e-4aaa-a052-70ef136cbd8f/volumes/kubernetes.io~projected/kube-api-access-z92tt:{mountpoint:/var/lib/kubelet/pods/8ef936d7-c04e-4aaa-a052-70ef136cbd8f/volumes/kubernetes.io~projected/kube-api-access-z92tt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~projected/kube-api-access-9r9x4:{mountpoint:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~projected/kube-api-access-9r9x4 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~projected/kube-api-access-9wd9r:{mountpoint:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~projected/kube-api-access-9wd9r major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~projected/kube-api-access-hrn2n:{mountpoint:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~projected/kube-api-access-hrn2n major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adfeeee0-f0b9-4317-8ada-11b67a839acf/volumes/kubernetes.io~projected/kube-api-access-h9h7m:{mountpoint:/var/lib/kubelet/pods/adfeeee0-f0b9-4317-8ada-11b67a839acf/volumes/kubernetes.io~projected/kube-api-access-h9h7m major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~projected/kube-api-access-5g2tx:{mountpoint:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~projected/kube-api-access-5g2tx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c29807b1-5429-4b5e-9f35-a85384476acb/volumes/kubernetes.io~projected/kube-api-access-kdfd5:{mountpoint:/var/lib/kubelet/pods/c29807b1-5429-4b5e-9f35-a85384476acb/volumes/kubernetes.io~projected/kube-api-access-kdfd5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~projected/kube-api-access-nbsr4:{mountpoint:/var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~projected/kube-api-access-nbsr4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:492 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4352351-b073-4b69-a7c3-f4f46fd6cc64/volumes/kubernetes.io~projected/kube-api-access-vb4nn:{mountpoint:/var/lib/kubelet/pods/d4352351-b073-4b69-a7c3-f4f46fd6cc64/volumes/kubernetes.io~projected/kube-api-access-vb4nn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~projected/kube-api-access-bqqtw:{mountpoint:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~projected/kube-api-access-bqqtw major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/alertmanager/9 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/config-reloader/6 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~empty-dir/config-out major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/kube-api-access-9snn4:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/kube-api-access-9snn4 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/tls-assets major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/config-volume major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/web-config major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~projected/kube-api-access-jh55x:{mountpoint:/var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~projected/kube-api-access-jh55x major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~projected/kube-api-access-bm7tx:{mountpoint:/var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~projected/kube-api-access-bm7tx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3c351b70fb4a387ee251288d50a7c3c600fa42e15e207a4f358d9427657cd44d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e6a8aeef7eb24d6391f5b63e209dff124fab5dd3bb477501d0bbb9ae88b6e07d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b4d016037f8d23338cff4ae7d685cb419bb57305d1b239967263eb95c2a274b6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/15f55f8718c24d8c57ddf4c09ae3034c623a2e2017609e83a9c0b449dbae7b4e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/8fb7ea72f4a05d6cdc91e5c4022bb70704ecb092ac8671b68df6530bbbe21c2c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b6a37c47fe8deac8234bd3ce0aece806d86eb0a03b47ae44616cae9ad068d6bf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5fa6aa98c06d947df300a9a63e8506bab0f0096e6f161df0bdd316b565ab8fd7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/05a80caa94ce960c66bb4f75562f82731d7bc41e20ac38d3b2fb1f41d242e193/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/7a5416f904a00a40350cee1da010a4661f3626499e0915b1da44f41929fdb297/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c3189d8068be89751a1a5d04e54d10196f084c637c3f76b3d74292ce5db1b6d8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e3b8c70133a87e8c98254e13b6b784295a875fc924a303c0245de99df5389970/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0ff8e7aee033c7c878442e69ebc3280cfc731a6fd0496023b76c54e2cf08d303/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9015cab70ac0c954d87bed6eca99f40591f3660e6a46ab98cb6552a5e6297668/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e1929702a672dc7a83be8e314233e4f59db85d34e3c84f8b515032b5a8580f50/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/786b25617b36cdf7b2062549330ddf8527c5b7a4d2e8b59e6da8019e7e8e09bc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/0f65deb3220514465dfc410e00263785173b51b91b61de161f0b7faceda46dfa/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/ac5abc4082494b8000d2fd6f70ef7b42a5945b80a51504a84babb756d9f97bda/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/db14060199ce559471323d5199588e69f2f29b261ed66df94137f792bc34f9e1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/805f0bc2be71a9bbc7593e596d4e760245c5aa1b9d5ab34fe34ed2cb34975f49/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0193398b49a6942ce17a3af38b09f03a142a0a30c18f587aaa494b758a02022d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c5f1e0af007860e07c1a989732e61721f6f9d014c615b7d32e5d94d12ce5f960/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/90671bd4c442cd705e0ff45f6dff5e83bca7a07401ca8f56d4d5461ac3fa17c2/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0b0ca89d88744bf52ef9e63adb3753d15ffbced2f12bbf66986a99ae05ce8081/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/92e72e3c3e9ed2b6ebd458713fd7d6f5b9ca6e97dbecc2d93c5b50067182ed79/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f26ae8d41f01c45aee99027fa2764745ba4a97bdda9f6addd801e8cad3182cb9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/7715b79a1d351afa588d4a16c56a6376a9ad4fc7ea1684e35580508f1ba1a2ec/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ee9a808fc215882fc89e806bfb651c401deba5d02922953864c6d4a2cfc6626b/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/dc41a17addb3d6bd333650d8468cb47d4c854821c237853faeebd0b881945158/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/cdcb965c7dca59900a87bff1440593884f5cd682ae228cf4a66314116f9dd85c/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/81b5bb55e2208ed0666b7731af139fd1463893c518de4a6eb1f281194ad3fd8d/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9633a78b207b0879b8375d4f7dd85d51ea1500f71f284d8af2b873340f3b5585/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/15090293d4cb4ad1a22d81866b064a69251ddfd59f9bf68983790f94faf74890/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/7ea3b91d0f7d2934b0679f933545bb0e82ff9b224d82bebe845d5e9f990af1b4/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/6cd4c5557b0b76ccf16abff574ae9b8c994aa97feb64c4bfb6fa2b1bfb46c903/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/24596d573a886089e549998f8d582c74991bba4c9580cedade1f268d022ef311/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/b47f2f546aa2c3c6feee163b1ff4dc6b404d8a765ccb168846972ea753bf377f/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/6d8a94c7a24daddc6c25e5e612951a42c070bb4b9f7a07058bf5ca1e137c9dd8/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/4d712152c546a08b1844c6d62aae72218401b5b52a693b12224e1b328faf8896/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/76aeec17524ace35083189eaeca28f7ae70f78c7ed4b6d5f3fb280f8cd772dc3/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/5dae0d7f91e6f0b63ffd873ca507b4417d91cfa61e0b4cc41455eb591f95b2e0/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/e6a13b5ccbdda3d994fdae4a63beb102c080df7e081ec3a57cc2a4d2ab39cdf3/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/9b8865ad854d9c9a7281b6fe21a809339b15a5aa71c02191b6091b1fe0b19b44/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/a2f1d9f4978fd5e07860daae99f1f1836e5658e1fce869b5c3690af7acaa67bb/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/15099178c80a4879a440aede162c4130f6bd1bc8383649b091c8f0291cae0a25/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/91735047a6464fcb987c7d52cfb8fc3447e0415a0cf0b56f3295bfea3b963fdf/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/4438b5a78b3276db6efaa9037f203a2dec5e16c4e56d397ad15f813bca596742/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/2cb0e960b9f4737f62cc6c839c7de3ccb0683b57e08f97c33e3fa98ed1767e6b/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/2fb3b50de4696816cdf51902340c7451e2a5dba98bf1942ced5b98ca7ded27d7/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/abb9cbbf0a6649b0e5ccb77132ce08cd49d9eb53627b08a64030f07508ac5568/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/3efa4d0554e867a06901222cb7b3b65fd17dd961bdec8d1e2d6af6478a2d0f71/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/d06c836d91d79636f5161a971973327f11aad79c09df0d34d34d61152a112a92/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/2ca2f9c662b29644c87839e57e678fb9f72e8c0cefd7949ff9697c07eb7b6710/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/2c18c875e73124818a6a25da82be2cc96b55b92bf42bb183b14bb12b9e7dec58/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/3f83ef89157225964bc63840ae73adbc1f2d3f597accb2416304208ef8d39b2f/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/3a2cea6774a0fc48453ddddfe92562376829f8aa6045db510542fa3df30fcef4/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/2644f60b585de9f638f1edaad7ba62e15259d3a0021df48ff6dc32fc74a502e2/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/03b02bcbee55c6d85b6f2b30be7d1766ea40fd34069ff2526c08a263b403dca2/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/ea97564af1a04e300cb2c5c31b91245fdba4b399e392e36e0dae9c954723d0ea/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/72a8f33107b224bd32597dbc812923cf916625decb8d46e3ad7d3c93f4a796a8/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/70b8119ef149977b2e247bfaa4a6b5d657cd049aa95451141c9966a15141dcb5/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/3300eeef86d82f89a9ccef0a06c72a53ff860670946be726a8698c4e50a41ebf/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/3192274cce20e615d09e80cb110277beef1a07f7d9ef9cc27a7004c46ae1cefb/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/86f739eaa6ea11618568cc4653cea6a73bfda53b0194861e5667038ea2062615/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/90cf653856ee09c8cb5d6b8df2e9e892f3b4f87b0960c9f62807f0801d6c7ff1/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/ca94bee0ce7c499ee1c6b950881a4ffaf9c54dcd87e6d6a0077bcfefd5d2393f/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/cbc2f4ae5b98c95eb8ffdf18681c43f240b63752538ed659880428869e23ae43/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/3dc28f786b0e3b95d892c0eee0805985f09d4862a97241e47f4c27b1967df747/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/42d24808dba4718989b9cd6da0dd78519f53c01e947ed7b9f129fcf2423e9e41/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/cc29b397952c65eac1cbc632f55700f23dfe7a3f11e822dee62ed063d97c4abf/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/f17ac37036f439ca78130c3498a9c0372421528c609bfb5a669ba1e698eaa8cf/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e8a7c581b08e52a939e22ee1bfeccf74c26a8d39bc84aeda9d729355e1666829/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/12f024c05cd7152ce27b943a4d29d5015d71d1f4f70814b5129c3b0d644407a0/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/1f4f07624336e61a33d81c20e4940ee2ae656cf00157ab50351dfd1cfca67d0a/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/fd52f0512787223341e8d66b0696e2c286b4ebb68f5d24f3afc42b4e9ec00098/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/b0ed803bdfa7f4a84d55547dcd45c8b0045a446ca597c4227bc2c1d3f3861d2c/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/09f4017b67b345a6b0d15b2bb197cbc3cb05a1cfe9561c6a1f94adae775dea15/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/b4e85c383929659ef20ed466adfce2080c00dac71c1463f54f33ccc8c49e8968/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/d281ccb88831d6a721535e4170c028d8c961969a2e36be77ef9c36c5f43bb670/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-529:{mountpoint:/var/lib/containers/storage/overlay/c4a4604616dedb2d55e1519119b86b5e9f463689a25a69e75a664f171cf8ad59/merged major:0 minor:529 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c22d58d4e3f78662b7ef0c3a122a6da385a0d4c1e4b7705eb488d3bf48c922f7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/7062718318de64eb9094c7f6868ee1ac78fb5ed8e8c97ede2146c56852530edc/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/590e8a358dc5a5347c0d2a71fccd8139cc9d6995aea5159cd2fae2ac969f0e54/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4f8db264ff6131affb8f8718d191fc7146ba8262547a807afebad3ddb469ebf6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e4710bd27d6af8bf1c58acf7ceffaa24ee407e4ceb3898c3d55146f67eef43af/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/cba9fc6f3367ccc831ef348799fd29bf867a6417faadcae36dc3d706cd6e4c1d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/286ffb850a962326796597bbd51588276332f5fb3ec347d6d22b2b5d15fc47ba/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/46761a71b5c49eaa73eb09c2ffa0beeafd46308d50c848dc36e1426d8f701c5e/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/15e819a0a269d3b3a9f476fe30a8ed59a95ffa5404a29bb60f6e131efe7592b0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.746892    6033 manager.go:217] Machine: {Timestamp:2026-10-30 23:42:27.74629378 +0000 UTC m=+0.093277671 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099898 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a59bd40b1e5d81fb8bc75ab065fdb SystemUUID:ec2a59bd-40b1-e5d8-1fb8-bc75ab065fdb BootID:98b04ec4-881a-4eac-980a-08376b815cbc Filesystems:[{Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~projected/kube-api-access-p9qnn DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:429 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:433 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:442 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~projected/kube-api-access-5g2tx DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:440 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~projected/kube-api-access-hrn2n DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c54f138db6cdbdf10dded963c8334d6d2c816642c9eb357372dd86f686c7fd1c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:438 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f/userdata/shm DeviceMajor:0 DeviceMinor:452 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c29807b1-5429-4b5e-9f35-a85384476acb/volumes/kubernetes.io~projected/kube-api-access-kdfd5 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e90b3b72585fe4db1b4c4ad10abf66a48b8c57c725f08dce8eed63bc43e151d3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b23556783a9e1c7c32bb5993c6b657fc133a46de698baa1d9472c86d3af41ef/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/494befaf7250b9d6f8ea542f44085f72fd02527f2542b537ab76568a9867386f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/volumes/kubernetes.io~projected/kube-api-access-bqqtw DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9019704-e296-412f-86a7-248fdcb0cfb0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d2a3b8acba6c87f34818738e443a9b16cd15a88620d80da2a19c36200e5722b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:492 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b24170b6dce43c5e4c7d71385287f67e33b47b7daf009ddd3d24b334ebfcc80e/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:435 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/kube-api-access-9snn4 DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55cead224e0b1fa33b6cec3cae387a242ae43acccd3234fcfca811683764604e/userdata/shm DeviceMajor:0 DeviceMinor:502 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e530137f-bb24-4651-b3bf-b634487b3fbc/volumes/kubernetes.io~projected/kube-api-access-jh55x DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d30058129d921f06ec383e0e3154b5828266c2f10b1c6f0084c871b8c12a58a8/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:441 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-529 DeviceMajor:0 DeviceMinor:529 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4352351-b073-4b69-a7c3-f4f46fd6cc64/volumes/kubernetes.io~projected/kube-api-access-vb4nn DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ef936d7-c04e-4aaa-a052-70ef136cbd8f/volumes/kubernetes.io~projected/kube-api-access-z92tt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~projected/kube-api-access-9r9x4 DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:439 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c8d1476a46f99b9a5834120a4c62ca5a4f310c4c4fa72e0110b4f9dfa48b2b8/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:319 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:437 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:359 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/681ea7db-6b38-46c2-91d2-80e27ef3c1af/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:496 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b9656a7-5042-408c-88a3-b607490fa46b/volumes/kubernetes.io~projected/kube-api-access-bpwnh DeviceMajor:0 DeviceMinor:497 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf0f293d24fc5eb0d1200ad942be46b484010c187c19829a019a9c4b42cf01d2/userdata/shm DeviceMajor:0 DeviceMinor:498 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78fd1a86a5dc3bdc9beeef54c194592293917c3d64d96f4108fb08d5a1a4e16a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6123bd6ef849725342e8a9f1c879482ce2419d07886bf958727990c8682637cf/userdata/shm DeviceMajor:0 DeviceMinor:500 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/790c4c63-efba-46c4-b27f-db092797bca3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12daebb8ba41831b321bd5a7d89fc6511dfc32c5b4f6c240447e69e831f1cc7f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a58a00569883d6f2819999a88ead90e1799e5c09bef110f08c06adf88d6bdc39/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~projected/kube-api-access-sxmm5 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~projected/kube-api-access-rtn49 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ca967018-daec-4727-aeea-4227b319d56c/volumes/kubernetes.io~projected/kube-api-access-nbsr4 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bba839b2d3fcfde06941cecff00bb62d67c2a6f489bf8600fe50d6f593f7d5a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9fd34b7a23a8f2c940b0715594b35801382622ce7d1fa9003ef5055126a34f7/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2276a6da-3505-4ad6-a190-632d6586d916/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~projected/kube-api-access-bm7tx DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d346c8e92a8568fe02ab3fd160a53f0bf5f1b7602b7f087e3c1316b8fb455242/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed2164b-7445-41c3-a449-bc2b0168ba72/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7eb21706-15cd-4498-8e41-32d421e417e9/volumes/kubernetes.io~projected/kube-api-access-s4p5c DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5a8f8b0ff615465cbb4209dfc602539fc4084a085c612c3dd75303d0a66be2c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~projected/kube-api-access-9wd9r DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff47f4f6055c49fbfd43c123c3635b3e957705116b8ef507f3d610f933a1470a/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/389e6e7e-8415-4d41-8583-3bae15af7e96/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34d59f9cd40802dac81cf9092041820d1c32647d36289d7de953de8879c27e77/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0cc93e9-3bb7-4224-9575-d26cf94807c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26c77996-97b5-4656-9ed8-4d27ba35f8ab/volumes/kubernetes.io~projected/kube-api-access-n999s DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3431afb6-ad2e-44c3-a43b-2e11393e6523/volumes/kubernetes.io~projected/kube-api-access-v4bl4 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc0416c-9ced-475c-8022-97cac5f04340/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6c3d6cbcc1ffc9fcdf901ebe2852fe89a31b6a5500bf2d43896134a97fd024a/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes/kubernetes.io~projected/kube-api-access-lz8wn DeviceMajor:0 DeviceMinor:443 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/859cdafc-83b2-4a33-b5f1-b2bf87a8405c/volumes/kubernetes.io~projected/kube-api-access-gr2h6 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adfeeee0-f0b9-4317-8ada-11b67a839acf/volumes/kubernetes.io~projected/kube-api-access-h9h7m DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1c8d1476a46f99b MacAddress:ce:13:51:d1:75:69 Speed:10000 Mtu:8901} {Name:494befaf7250b9d MacAddress:06:bf:44:99:41:71 Speed:10000 Mtu:8901} {Name:55cead224e0b1fa MacAddress:46:74:70:0d:65:61 Speed:10000 Mtu:8901} {Name:6123bd6ef849725 MacAddress:36:51:00:16:3a:9d Speed:10000 Mtu:8901} {Name:8f4e727345b2de1 MacAddress:4a:49:d5:e6:1e:e3 Speed:10000 Mtu:8901} {Name:a58a00569883d6f MacAddress:1e:c0:63:7c:90:c9 Speed:10000 Mtu:8901} {Name:b24170b6dce43c5 MacAddress:5a:23:93:2b:a7:39 Speed:10000 Mtu:8901} {Name:b5a8f8b0ff61546 MacAddress:ea:25:83:6d:3c:0d Speed:10000 Mtu:8901} {Name:b9fd34b7a23a8f2 MacAddress:5a:30:48:48:9b:e2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d4:14:cd:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:52:21:e6:3b:1f Speed:0 Mtu:8901} {Name:cf0f293d24fc5eb MacAddress:d2:57:e8:a9:ad:e7 Speed:10000 Mtu:8901} {Name:d30058129d921f0 MacAddress:92:8e:86:cf:0b:e6 Speed:10000 Mtu:8901} {Name:d346c8e92a8568f MacAddress:2a:62:1a:8d:b7:59 Speed:10000 Mtu:8901} {Name:dbf1401b61f4b15 MacAddress:32:90:e7:77:d6:be Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d4:14:cd:33 Speed:0 Mtu:9001} {Name:ff47f4f6055c49f MacAddress:22:74:10:91:70:23 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:3b:54:08:59:e9 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:84:4a:9d:98:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.747456    6033 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.747518    6033 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.747893    6033 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.747926    6033 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-66.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748175    6033 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748188    6033 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748208    6033 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748237    6033 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748383    6033 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748602    6033 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748744    6033 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748784    6033 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748806    6033 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748822    6033 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748837    6033 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:42:27.749301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.748850    6033 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:42:27.750419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.749834    6033 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:42:27.750419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.749849    6033 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:42:27.751415 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.751397    6033 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:42:27.755444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755424    6033 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:42:27.755444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755443    6033 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:42:27.755589 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755558    6033 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 23:42:27.755812 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755797    6033 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:42:27.755861 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755840    6033 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:42:27.755961 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755941    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:42:27.755961 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755963    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:42:27.755961 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755970    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755975    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755982    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755991    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.755999    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756005    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756012    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756018    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756052    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756067    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756093    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:42:27.756213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.756098    6033 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:42:27.760276 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.760259    6033 server.go:1263] "Started kubelet"
Oct 30 23:42:27.760806 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.760733    6033 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:42:27.760806 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.760798    6033 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:42:27.761110 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.761075    6033 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:42:27.760855 ip-10-0-139-66 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:42:27.761997 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.761168    6033 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:42:27.767432 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.767407    6033 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:42:27.776583 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.776562    6033 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:42:27.776838 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.776758    6033 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:42:27.779474 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:27.778083    6033 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:42:27.779474 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.778149    6033 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:42:27.779474 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.778157    6033 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:42:27.779474 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.778396    6033 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:42:27.779907 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.779602    6033 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:36:17 +0000 UTC" deadline="2028-07-31 01:45:13.981971972 +0000 UTC"
Oct 30 23:42:27.779907 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.779644    6033 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15338h2m46.202337814s"
Oct 30 23:42:27.781501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781299    6033 factory.go:55] Registering systemd factory
Oct 30 23:42:27.781501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781319    6033 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781489    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781523    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781538    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781550    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781562    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781575    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781586    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781599    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b9656a7-5042-408c-88a3-b607490fa46b" volumeName="kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781613    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781625    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb21706-15cd-4498-8e41-32d421e417e9" volumeName="kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config" seLinuxMountContext=""
Oct 30 23:42:27.781643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781637    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781650    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfeeee0-f0b9-4317-8ada-11b67a839acf" volumeName="kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781662    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781674    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781714    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781727    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2276a6da-3505-4ad6-a190-632d6586d916" volumeName="kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781741    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781754    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="681ea7db-6b38-46c2-91d2-80e27ef3c1af" volumeName="kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781765    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781779    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781792    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e" volumeName="kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781805    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781819    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29807b1-5429-4b5e-9f35-a85384476acb" volumeName="kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781831    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781843    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781872    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781885    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781899    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781729    6033 factory.go:153] Registering CRI-O factory
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781916    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2276a6da-3505-4ad6-a190-632d6586d916" volumeName="kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781930    6033 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781995    6033 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782019    6033 factory.go:103] Registering Raw factory
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782049    6033 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.781929    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c77996-97b5-4656-9ed8-4d27ba35f8ab" volumeName="kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782072    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782088    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e" volumeName="kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782102    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782114    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cc93e9-3bb7-4224-9575-d26cf94807c7" volumeName="kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782125    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4352351-b073-4b69-a7c3-f4f46fd6cc64" volumeName="kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782139    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782151    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782163    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782175    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls" seLinuxMountContext=""
Oct 30 23:42:27.782174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782185    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e" volumeName="kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782198    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782211    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782223    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782238    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2276a6da-3505-4ad6-a190-632d6586d916" volumeName="kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782250    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782262    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782273    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782285    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782297    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782310    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca967018-daec-4727-aeea-4227b319d56c" volumeName="kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782322    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782334    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3431afb6-ad2e-44c3-a43b-2e11393e6523" volumeName="kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782345    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782357    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782369    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790c4c63-efba-46c4-b27f-db092797bca3" volumeName="kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782380    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782404    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782418    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859cdafc-83b2-4a33-b5f1-b2bf87a8405c" volumeName="kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782427    6033 manager.go:319] Starting recovery of all containers
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782429    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782444    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782457    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782468    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb21706-15cd-4498-8e41-32d421e417e9" volumeName="kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782480    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782492    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adfeeee0-f0b9-4317-8ada-11b67a839acf" volumeName="kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782504    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782516    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 23:42:27.788905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.782528    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e530137f-bb24-4651-b3bf-b634487b3fbc" volumeName="kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert" seLinuxMountContext=""
Oct 30 23:42:27.792633 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792600    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c77996-97b5-4656-9ed8-4d27ba35f8ab" volumeName="kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792636    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3431afb6-ad2e-44c3-a43b-2e11393e6523" volumeName="kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792658    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3431afb6-ad2e-44c3-a43b-2e11393e6523" volumeName="kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792673    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb21706-15cd-4498-8e41-32d421e417e9" volumeName="kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792685    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792703    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cc93e9-3bb7-4224-9575-d26cf94807c7" volumeName="kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 23:42:27.792715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792715    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9019704-e296-412f-86a7-248fdcb0cfb0" volumeName="kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792732    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2cd805f-b97f-44d4-87f2-ddb1d8ad770e" volumeName="kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792746    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792760    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="790c4c63-efba-46c4-b27f-db092797bca3" volumeName="kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792776    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e" volumeName="kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792789    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792805    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2cd805f-b97f-44d4-87f2-ddb1d8ad770e" volumeName="kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792816    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2cd805f-b97f-44d4-87f2-ddb1d8ad770e" volumeName="kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792829    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792848    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792859    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792875    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c29807b1-5429-4b5e-9f35-a85384476acb" volumeName="kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792887    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e530137f-bb24-4651-b3bf-b634487b3fbc" volumeName="kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792899    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2cd805f-b97f-44d4-87f2-ddb1d8ad770e" volumeName="kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792916    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c77996-97b5-4656-9ed8-4d27ba35f8ab" volumeName="kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792929    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="389e6e7e-8415-4d41-8583-3bae15af7e96" volumeName="kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792945    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792957    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb21706-15cd-4498-8e41-32d421e417e9" volumeName="kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792971    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef936d7-c04e-4aaa-a052-70ef136cbd8f" volumeName="kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.792988    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc0416c-9ced-475c-8022-97cac5f04340" volumeName="kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793000    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cc93e9-3bb7-4224-9575-d26cf94807c7" volumeName="kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 23:42:27.793008 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793018    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9019704-e296-412f-86a7-248fdcb0cfb0" volumeName="kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793048    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793061    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793078    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5200d5bb-5437-456e-93bb-3c95d0e19d1d" volumeName="kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793091    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eb21706-15cd-4498-8e41-32d421e417e9" volumeName="kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793110    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859cdafc-83b2-4a33-b5f1-b2bf87a8405c" volumeName="kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793122    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0cc93e9-3bb7-4224-9575-d26cf94807c7" volumeName="kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793134    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca967018-daec-4727-aeea-4227b319d56c" volumeName="kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793152    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b" volumeName="kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793165    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2276a6da-3505-4ad6-a190-632d6586d916" volumeName="kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793182    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3431afb6-ad2e-44c3-a43b-2e11393e6523" volumeName="kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793194    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed2164b-7445-41c3-a449-bc2b0168ba72" volumeName="kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793206    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ef936d7-c04e-4aaa-a052-70ef136cbd8f" volumeName="kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793224    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9019704-e296-412f-86a7-248fdcb0cfb0" volumeName="kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793236    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793253    6033 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" volumeName="kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793265    6033 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:42:27.794025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.793274    6033 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:42:27.816530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.816334    6033 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:42:27.821129 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.821088    6033 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:42:27.821129 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.821114    6033 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:42:27.821345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.821140    6033 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:42:27.821345 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:27.821184    6033 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:42:27.866771 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.866688    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870" exitCode=0
Oct 30 23:42:27.878181 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:27.878150    6033 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:42:27.878368 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.878255    6033 generic.go:356] "Generic (PLEG): container finished" podID="fbc66296f1ac47a99ab1c1b04d380e57" containerID="70dd65ac2e645b85f622505d8baacdb0ccfd07e4cd74af56e0fc941c28c8e61f" exitCode=0
Oct 30 23:42:27.921472 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:27.921304    6033 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 23:42:27.949116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949066    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="199312e4f35cb914b3d2a7c8c456849e92ec184345e61a59c533308754298bc1" exitCode=0
Oct 30 23:42:27.949116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949106    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="5550bd646908414683f4ebb9f1686226202c74666c31a1a170b2e2adc21031b0" exitCode=0
Oct 30 23:42:27.949116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949116    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="545e56bbd97d7375b89550f8389cfc3264ece0b90344c22feccdfbe655b0775f" exitCode=0
Oct 30 23:42:27.949116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949126    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="fae8dba104bbdc80e505562dba67395fdd36abe0b0c87ff2dbfe0de6ad446ed2" exitCode=0
Oct 30 23:42:27.949439 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949134    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="9fd57bbb054d0ab24ca4119744ced854c57ee4c930a1649261e91170726514f0" exitCode=0
Oct 30 23:42:27.949439 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.949142    6033 generic.go:356] "Generic (PLEG): container finished" podID="3431afb6-ad2e-44c3-a43b-2e11393e6523" containerID="8b3ae473cc955cc134f7dc61f0488366941db0dc645ac8a09af3a211b59f0866" exitCode=0
Oct 30 23:42:27.952633 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.952609    6033 manager.go:324] Recovery completed
Oct 30 23:42:27.953334 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.953314    6033 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 23:42:27.953519 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.953335    6033 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:42:27.953519 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.953451    6033 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:42:27.960990 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.960961    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd" exitCode=0
Oct 30 23:42:27.963852 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.963796    6033 generic.go:356] "Generic (PLEG): container finished" podID="9cc0416c-9ced-475c-8022-97cac5f04340" containerID="a1f9c76a494c6fc3e21cf47e9e581f1e2cf049aff222b764791ea6c8abc20f3f" exitCode=0
Oct 30 23:42:27.978499 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:27.978461    6033 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-66.us-west-2.compute.internal\" not found"
Oct 30 23:42:27.983892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.982919    6033 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:42:27.985755 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.985730    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:42:27.985862 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.985769    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:42:27.985862 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.985787    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:42:27.989931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.989907    6033 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:42:27.989931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.989927    6033 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:42:27.990067 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.989940    6033 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:42:27.990112 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.989948    6033 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:42:27.990323 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990305    6033 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 23:42:27.990375 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990324    6033 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 23:42:27.990375 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990349    6033 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 23:42:27.990375 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990358    6033 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 23:42:27.990375 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990366    6033 policy_none.go:50] "Start"
Oct 30 23:42:27.990375 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990377    6033 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:42:27.990603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990389    6033 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:42:27.990603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990557    6033 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 23:42:27.990603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990569    6033 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 23:42:27.990603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.990578    6033 policy_none.go:44] "Start"
Oct 30 23:42:27.993619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.993513    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:42:27.993619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.993551    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:42:27.993619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.993568    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:42:27.996871 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.996828    6033 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:42:27.998714 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:27.998690    6033 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:42:28.004676 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.004657    6033 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:42:28.005464 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:28.004693    6033 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:42:28.005600 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.005468    6033 server.go:87] "Starting device plugin registration server"
Oct 30 23:42:28.005829 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.005810    6033 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:42:28.005932 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.005828    6033 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:42:28.006075 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.006057    6033 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:42:28.006233 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.006167    6033 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:42:28.008014 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.007993    6033 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:42:28.008122 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.008017    6033 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:42:28.008167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.008082    6033 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:42:28.006198499Z","UUID":"dc3f722b-7b9c-4a3f-9f8e-81b65b34a079","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:42:28.008356 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:28.008337    6033 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:42:28.009784 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.009762    6033 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:42:28.009921 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.009805    6033 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:42:28.045860 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.045639    6033 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:42:28.106888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.106852    6033 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:42:28.108123 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.108091    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:42:28.108254 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.108133    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:42:28.108254 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.108150    6033 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-66.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:42:28.108254 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.108220    6033 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.122351 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.122241    6033 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"]
Oct 30 23:42:28.123242 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123218    6033 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.123369 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123191    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerStarted","Data":"585557e03a1ffe9a64617ca26c9ad5fd0bb6ed28b461d599fd8908aeb59ab3bd"}
Oct 30 23:42:28.123369 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123356    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerDied","Data":"70dd65ac2e645b85f622505d8baacdb0ccfd07e4cd74af56e0fc941c28c8e61f"}
Oct 30 23:42:28.123471 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123391    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"fbc66296f1ac47a99ab1c1b04d380e57","Type":"ContainerStarted","Data":"0fc704fba85bdaaf0c60439e3c27aeb3daab96ba92df79a8b7fc77a7c20023a4"}
Oct 30 23:42:28.123471 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123454    6033 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.123571 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123455    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"4e63feea4e10327df2e22d81d9a9cb4a","Type":"ContainerStarted","Data":"fcf68e33efccaf4680616564109321fef23f222d5454626024c9aad0c02fac93"}
Oct 30 23:42:28.123571 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.123553    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal" event={"ID":"4e63feea4e10327df2e22d81d9a9cb4a","Type":"ContainerStarted","Data":"1ad2806838e49d5ffdc7c9c19191ebe370f27eb78988933590dd4a204a92f1ad"}
Oct 30 23:42:28.182848 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.182809    6033 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:42:28.183059 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:28.182876    6033 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.183364 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.183342    6033 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:42:28.183432 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:28.183394    6033 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.210709 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.210675    6033 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.210890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.210761    6033 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.254992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.254955    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.255183 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.254999    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.255183 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.255025    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.355909 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.355867    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.356109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.355917    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.356109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.355946    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.356109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.355998    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e63feea4e10327df2e22d81d9a9cb4a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"4e63feea4e10327df2e22d81d9a9cb4a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.356109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.355996    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.356109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.356088    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbc66296f1ac47a99ab1c1b04d380e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" (UID: \"fbc66296f1ac47a99ab1c1b04d380e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.750498 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.750459    6033 apiserver.go:52] "Watching apiserver"
Oct 30 23:42:28.898897 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.898860    6033 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:42:28.899791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.899750    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86dfbdff78-j4gh6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal","openshift-monitoring/telemeter-client-54885d6898-mpk4r","openshift-network-operator/iptables-alerter-dp969","openshift-dns/node-resolver-lbzmk","openshift-insights/insights-runtime-extractor-b9lzs","openshift-multus/multus-z85qc","openshift-dns/dns-default-gm9ps","openshift-ingress-canary/ingress-canary-8lbnc","openshift-cluster-node-tuning-operator/tuned-gfqft","openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv","openshift-monitoring/prometheus-operator-9cd89b74c-847hn","openshift-network-diagnostics/network-check-target-dkcdb","openshift-ovn-kubernetes/ovnkube-node-htb47","kube-system/konnectivity-agent-w2k95","openshift-monitoring/prometheus-k8s-0","openshift-console/downloads-6dd789c8b7-mxk2r","openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr","kube-system/global-pull-secret-syncer-fjfcx","openshift-image-registry/node-ca-q6c68","openshift-multus/network-metrics-daemon-h6w9m","kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt","openshift-monitoring/node-exporter-vvtvb","openshift-multus/multus-additional-cni-plugins-lzljn","openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:28.953171 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953134    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953168    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953397    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953415    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953345    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953342    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953451    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953195    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953321    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:42:28.953505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953368    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953581    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953624    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953629    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953742    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953831    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-72946\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953857    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zwz57\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953867    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bmqk5\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953902    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953914    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953959    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ksfp2\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953978    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.953985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953989    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnlbs\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.953993    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hcpbw\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954046    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954071    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954072    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954262    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954335    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954359    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954365    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954430    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.954542 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954544    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954590    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-554bl\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954626    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hnph4\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954703    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zg485\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954771    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954851    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-294w8\""
Oct 30 23:42:28.954978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.954925    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:42:28.970978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.970954    6033 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.971164 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.971102    6033 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:28.982379 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:28.982350    6033 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:42:29.026475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.026406    6033 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:42:29.026475 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:29.026471    6033 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:29.026969 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.026949    6033 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:42:29.027075 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:29.027006    6033 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-66.us-west-2.compute.internal"
Oct 30 23:42:29.050173 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050140    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:42:29.050373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050195    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 23:42:29.050373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050235    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 23:42:29.050373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050310    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 23:42:29.050546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050400    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 23:42:29.050546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050310    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:42:29.050652 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050549    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:42:29.050652 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050598    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:42:29.050746 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050678    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:42:29.050746 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050698    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:42:29.050746 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050729    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 23:42:29.050902 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050761    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:42:29.050902 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050819    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:42:29.050902 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.050860    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:42:29.051429 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.051328    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:42:29.051589 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.051568    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:42:29.051699 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.051680    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 23:42:29.051846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.051832    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:42:29.051995 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.051955    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:42:29.052094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.052015    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:42:29.060619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060589    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060629    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060653    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060677    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060710    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060737    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.060778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060761    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbsr4\" (UniqueName: \"kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060786    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060812    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060835    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060859    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060895    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060936    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060970    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.060994    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061018    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061069    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.061090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061089    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ed2164b-7445-41c3-a449-bc2b0168ba72-metrics-client-ca\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061098    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z92tt\" (UniqueName: \"kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061136    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061160    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061185    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061209    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061412    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-config\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061457    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061505    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-daemon-config\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061516    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2h6\" (UniqueName: \"kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061541    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.061595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061569    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061594    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061634    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061656    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061689    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061715    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061750    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061777    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061800    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061928    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061956    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.061960    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2276a6da-3505-4ad6-a190-632d6586d916-data-volume\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062006    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062061    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062089    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.062124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062115    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062139    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062161    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062186    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062210    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062235    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062261    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qnn\" (UniqueName: \"kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062287    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062322    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062346    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062369    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062403    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4nn\" (UniqueName: \"kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062402    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-tmp-dir\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062560    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062590    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062613    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.062747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062638    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.063516 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062823    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-tmp-dir\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.063516 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.062874    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.063516 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063382    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2276a6da-3505-4ad6-a190-632d6586d916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.063516 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063493    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063569    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c29807b1-5429-4b5e-9f35-a85384476acb-iptables-alerter-script\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063675    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc93e9-3bb7-4224-9575-d26cf94807c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063759    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed2164b-7445-41c3-a449-bc2b0168ba72-tmp\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063833    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-textfile\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063875    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtn49\" (UniqueName: \"kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.063920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.063903    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmm5\" (UniqueName: \"kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.064249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064101    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.064343 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064324    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/adfeeee0-f0b9-4317-8ada-11b67a839acf-tmp\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:29.064395 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064332    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.064745 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064725    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.064844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064755    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.064844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064783    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqtw\" (UniqueName: \"kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.064844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064810    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.064844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064836    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064862    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064888    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064910    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064925    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064945    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064971    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.064991    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7tx\" (UniqueName: \"kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.065018 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065015    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.065378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065082    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.065378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065108    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-config-volume\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.065378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065118    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrn2n\" (UniqueName: \"kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.065378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065143    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.065378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065243    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.065643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065423    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-tuned\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.065643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.065634    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca967018-daec-4727-aeea-4227b319d56c-metrics-certs\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:29.066278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066245    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n999s\" (UniqueName: \"kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.066460 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066279    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.066460 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066305    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.066591 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066508    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.066846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066570    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066840    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066918    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066936    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-serviceca\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066941    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.066961    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.067338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.067260    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.070590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.068298    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-serving-certs-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.070590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.068494    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cni-binary-copy\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073738    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh55x\" (UniqueName: \"kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073815    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073859    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073895    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073930    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.073978    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074013    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074076    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074129    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074207    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074234    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074267    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074322    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074389    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074397    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074431    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074440    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074587    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074620    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074627    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074672    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.074830 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.074707    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075315    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075359    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075411    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075450    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075502    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075547    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075589    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075615    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075646    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075668    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075673    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075720    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075751    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075779    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075809    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075836    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075863    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075866    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-metrics-client-ca\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.075986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.075956    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd9r\" (UniqueName: \"kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076076    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076108    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9x4\" (UniqueName: \"kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076151    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076178    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfd5\" (UniqueName: \"kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076222    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076246    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076253    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/790c4c63-efba-46c4-b27f-db092797bca3-agent-certs\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076267    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076267    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076310    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076456    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g2tx\" (UniqueName: \"kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076495    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076523    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076574    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076611    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076640    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076668    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076772    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.076892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076807    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076915    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076951    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.076974    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/681ea7db-6b38-46c2-91d2-80e27ef3c1af-original-pull-secret\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077254    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077262    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-ovnkube-script-lib\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077296    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077503    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7eb21706-15cd-4498-8e41-32d421e417e9-ovn-node-metrics-cert\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077559    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077588    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077621    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077651    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h7m\" (UniqueName: \"kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077681    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077704    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077721    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077745    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.077772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077771    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077787    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077808    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077826    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077856    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077875    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077891    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077908    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4bl4\" (UniqueName: \"kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077924    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077946    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077965    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077980    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.077995    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078010    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078054    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078082    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078109    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078127    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078154    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078173    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078206    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p5c\" (UniqueName: \"kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078223    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.078425 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078238    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.079278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078475    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-metrics-client-ca\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.079278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078813    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.079278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.078939    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7eb21706-15cd-4498-8e41-32d421e417e9-env-overrides\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.079278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.079062    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9019704-e296-412f-86a7-248fdcb0cfb0-tmp\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.079456 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.079292    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/790c4c63-efba-46c4-b27f-db092797bca3-konnectivity-ca\") pod \"konnectivity-agent-w2k95\" (UID: \"790c4c63-efba-46c4-b27f-db092797bca3\") " pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:42:29.079816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.079792    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3431afb6-ad2e-44c3-a43b-2e11393e6523-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.079967 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.079945    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.080269 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.080040    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.098600    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.098767    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099235    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099281    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099429    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099447    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099524    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099558    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099566    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099605    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099650    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099656    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099744    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099745    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099778    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099785    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099793    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099792    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099824    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099829    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099850    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099861    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099866    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.099924    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100176    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100237    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100265    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100337    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100374    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100409    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xnb7d\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100457    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qbwc4\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100467    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7rnf\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100515    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100527    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fvkr7\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100561    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-w5vpr\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100609    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100632    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xzm7s\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100701    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r2zhx\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100766    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100833    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100889    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dkdfd\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100908    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100926    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100941    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ntvcp\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.100989    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101132    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4224x\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101205    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7qdml\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101306    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101409    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101492    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101511    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-mw499\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101585    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101656    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6s2q2\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101796    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-46rdvffkpcraf\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101835    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101873    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c266j\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.101942    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.102016    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 23:42:29.102116 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.102086    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.102246    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rm397jpo960s\""
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.102308    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.102411    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103154    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103443    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103457    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103666    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-metrics-tls\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103754    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.103898    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104092    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104122    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104255    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104311    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104313    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104441    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104427    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104543    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104683    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104847    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-federate-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.104915    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.105816    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.105825    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.105978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.105982    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106124    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106277    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106418    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106450    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106560    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106607    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-grpc-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106631    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106764    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0cc93e9-3bb7-4224-9575-d26cf94807c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106928    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.106964    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e530137f-bb24-4651-b3bf-b634487b3fbc-cert\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.107098    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.107147    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2276a6da-3505-4ad6-a190-632d6586d916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.107823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.107205    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.108371 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.108012    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/389e6e7e-8415-4d41-8583-3bae15af7e96-telemeter-client-tls\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.109345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.109325    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.109507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.109328    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7ed2164b-7445-41c3-a449-bc2b0168ba72-secret-thanos-querier-tls\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.110556 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.110533    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.110655 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.110567    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-tls\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.112842 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.112823    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.117908 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.117873    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4nn\" (UniqueName: \"kubernetes.io/projected/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kube-api-access-vb4nn\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.118020 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.117883    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbsr4\" (UniqueName: \"kubernetes.io/projected/ca967018-daec-4727-aeea-4227b319d56c-kube-api-access-nbsr4\") pod \"network-metrics-daemon-h6w9m\" (UID: \"ca967018-daec-4727-aeea-4227b319d56c\") " pod="openshift-multus/network-metrics-daemon-h6w9m"
Oct 30 23:42:29.118020 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.117955    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z92tt\" (UniqueName: \"kubernetes.io/projected/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-kube-api-access-z92tt\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.118932 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.118432    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"alertmanager-main-0\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:29.118932 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.118768    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwnh\" (UniqueName: \"kubernetes.io/projected/3b9656a7-5042-408c-88a3-b607490fa46b-kube-api-access-bpwnh\") pod \"network-check-target-dkcdb\" (UID: \"3b9656a7-5042-408c-88a3-b607490fa46b\") " pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:29.118932 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.118887    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2h6\" (UniqueName: \"kubernetes.io/projected/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-kube-api-access-gr2h6\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.119193 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.119170    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qnn\" (UniqueName: \"kubernetes.io/projected/389e6e7e-8415-4d41-8583-3bae15af7e96-kube-api-access-p9qnn\") pod \"telemeter-client-54885d6898-mpk4r\" (UID: \"389e6e7e-8415-4d41-8583-3bae15af7e96\") " pod="openshift-monitoring/telemeter-client-54885d6898-mpk4r"
Oct 30 23:42:29.119417 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.119400    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"prometheus-k8s-0\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:29.120217 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.120192    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmm5\" (UniqueName: \"kubernetes.io/projected/2276a6da-3505-4ad6-a190-632d6586d916-kube-api-access-sxmm5\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.120366 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.120339    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtn49\" (UniqueName: \"kubernetes.io/projected/7ed2164b-7445-41c3-a449-bc2b0168ba72-kube-api-access-rtn49\") pod \"thanos-querier-86dfbdff78-j4gh6\" (UID: \"7ed2164b-7445-41c3-a449-bc2b0168ba72\") " pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:29.120726 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.120702    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrn2n\" (UniqueName: \"kubernetes.io/projected/a0cc93e9-3bb7-4224-9575-d26cf94807c7-kube-api-access-hrn2n\") pod \"openshift-state-metrics-7bd7489957-6tgwv\" (UID: \"a0cc93e9-3bb7-4224-9575-d26cf94807c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6tgwv"
Oct 30 23:42:29.120860 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.120840    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqtw\" (UniqueName: \"kubernetes.io/projected/d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b-kube-api-access-bqqtw\") pod \"kube-state-metrics-5dc8d7d89b-8hddr\" (UID: \"d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-8hddr"
Oct 30 23:42:29.122931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.122911    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7tx\" (UniqueName: \"kubernetes.io/projected/f2cd805f-b97f-44d4-87f2-ddb1d8ad770e-kube-api-access-bm7tx\") pod \"dns-default-gm9ps\" (UID: \"f2cd805f-b97f-44d4-87f2-ddb1d8ad770e\") " pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:29.128639 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.128619    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n999s\" (UniqueName: \"kubernetes.io/projected/26c77996-97b5-4656-9ed8-4d27ba35f8ab-kube-api-access-n999s\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.128808 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.128788    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh55x\" (UniqueName: \"kubernetes.io/projected/e530137f-bb24-4651-b3bf-b634487b3fbc-kube-api-access-jh55x\") pod \"ingress-canary-8lbnc\" (UID: \"e530137f-bb24-4651-b3bf-b634487b3fbc\") " pod="openshift-ingress-canary/ingress-canary-8lbnc"
Oct 30 23:42:29.131585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.131562    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfd5\" (UniqueName: \"kubernetes.io/projected/c29807b1-5429-4b5e-9f35-a85384476acb-kube-api-access-kdfd5\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.131689 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.131667    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g2tx\" (UniqueName: \"kubernetes.io/projected/b9019704-e296-412f-86a7-248fdcb0cfb0-kube-api-access-5g2tx\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.131747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.131722    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9x4\" (UniqueName: \"kubernetes.io/projected/91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e-kube-api-access-9r9x4\") pod \"prometheus-operator-9cd89b74c-847hn\" (UID: \"91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-847hn"
Oct 30 23:42:29.131801 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.131755    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd9r\" (UniqueName: \"kubernetes.io/projected/9cc0416c-9ced-475c-8022-97cac5f04340-kube-api-access-9wd9r\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.133812 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.133791    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h7m\" (UniqueName: \"kubernetes.io/projected/adfeeee0-f0b9-4317-8ada-11b67a839acf-kube-api-access-h9h7m\") pod \"downloads-6dd789c8b7-mxk2r\" (UID: \"adfeeee0-f0b9-4317-8ada-11b67a839acf\") " pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:29.134535 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.134507    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4bl4\" (UniqueName: \"kubernetes.io/projected/3431afb6-ad2e-44c3-a43b-2e11393e6523-kube-api-access-v4bl4\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.135613 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.135595    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p5c\" (UniqueName: \"kubernetes.io/projected/7eb21706-15cd-4498-8e41-32d421e417e9-kube-api-access-s4p5c\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.178991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.178958    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.178991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.178995    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179020    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179091    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179113    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179137    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-root\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179170    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179190    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-etc-selinux\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179202    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-multus-certs\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179171    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179208    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179137    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179228    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-kubelet-config\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179240    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179255    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-registration-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.179262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179266    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179294    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179323    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179339    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-kubelet\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179345    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179346    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-netns\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179354    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-system-cni-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179378    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179388    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-device-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179406    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179430    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179410    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-sys\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179449    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179474    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179487    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-etc-kubernetes\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179483    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-cnibin\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179495    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-sys-fs\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179529    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-run\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179535    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179540    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ef936d7-c04e-4aaa-a052-70ef136cbd8f-hosts-file\") pod \"node-resolver-lbzmk\" (UID: \"8ef936d7-c04e-4aaa-a052-70ef136cbd8f\") " pod="openshift-dns/node-resolver-lbzmk"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179498    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-slash\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179572    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179587    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179604    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179629    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179642    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-var-lib-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179649    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-log-socket\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179656    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179708    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-netd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179716    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-systemd\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179738    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179768    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179793    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179825    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179853    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179877    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179882    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-hostroot\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179899    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179939    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179963    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179985    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180010    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.180005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180019    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-modprobe-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180056    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180079    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9cc0416c-9ced-475c-8022-97cac5f04340-node-exporter-wtmp\") pod \"node-exporter-vvtvb\" (UID: \"9cc0416c-9ced-475c-8022-97cac5f04340\") " pod="openshift-monitoring/node-exporter-vvtvb"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180083    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180109    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180131    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180148    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180169    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180209    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-conf\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180248    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180255    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-netns\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180283    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-run-k8s-cni-cncf-io\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180215    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180293    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-sys\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.179827    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/859cdafc-83b2-4a33-b5f1-b2bf87a8405c-host\") pod \"node-ca-q6c68\" (UID: \"859cdafc-83b2-4a33-b5f1-b2bf87a8405c\") " pod="openshift-image-registry/node-ca-q6c68"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180333    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29807b1-5429-4b5e-9f35-a85384476acb-host-slash\") pod \"iptables-alerter-dp969\" (UID: \"c29807b1-5429-4b5e-9f35-a85384476acb\") " pod="openshift-network-operator/iptables-alerter-dp969"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180338    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/681ea7db-6b38-46c2-91d2-80e27ef3c1af-dbus\") pod \"global-pull-secret-syncer-fjfcx\" (UID: \"681ea7db-6b38-46c2-91d2-80e27ef3c1af\") " pod="kube-system/global-pull-secret-syncer-fjfcx"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180345    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180373    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-etc-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180258    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3431afb6-ad2e-44c3-a43b-2e11393e6523-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzljn\" (UID: \"3431afb6-ad2e-44c3-a43b-2e11393e6523\") " pod="openshift-multus/multus-additional-cni-plugins-lzljn"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180375    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180379    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-systemd-units\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180337    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2276a6da-3505-4ad6-a190-632d6586d916-crio-socket\") pod \"insights-runtime-extractor-b9lzs\" (UID: \"2276a6da-3505-4ad6-a190-632d6586d916\") " pod="openshift-insights/insights-runtime-extractor-b9lzs"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180377    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-ovn\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180424    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-cni-bin\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180428    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-cnibin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180468    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180493    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180517    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-system-cni-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180527    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180552    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180575    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180596    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-bin\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180605    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180630    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180646    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysconfig\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180660    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180683    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-systemd\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180683    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180709    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180727    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180731    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-socket-dir-parent\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180749    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180772    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180781    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-run-openvswitch\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180795    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180817    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-multus-conf-dir\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180821    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180470    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-host-kubelet\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180900    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-kubernetes\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180943    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7eb21706-15cd-4498-8e41-32d421e417e9-node-log\") pod \"ovnkube-node-htb47\" (UID: \"7eb21706-15cd-4498-8e41-32d421e417e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.180962    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-lib-modules\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.181008    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-var-lib-kubelet\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.181065    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-etc-sysctl-d\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.181071    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d4352351-b073-4b69-a7c3-f4f46fd6cc64-socket-dir\") pod \"aws-ebs-csi-driver-node-c86kt\" (UID: \"d4352351-b073-4b69-a7c3-f4f46fd6cc64\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c86kt"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.181099    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c77996-97b5-4656-9ed8-4d27ba35f8ab-host-var-lib-cni-multus\") pod \"multus-z85qc\" (UID: \"26c77996-97b5-4656-9ed8-4d27ba35f8ab\") " pod="openshift-multus/multus-z85qc"
Oct 30 23:42:29.181765 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:29.181118    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9019704-e296-412f-86a7-248fdcb0cfb0-host\") pod \"tuned-gfqft\" (UID: \"b9019704-e296-412f-86a7-248fdcb0cfb0\") " pod="openshift-cluster-node-tuning-operator/tuned-gfqft"
Oct 30 23:42:31.955304 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:31.955268    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gm9ps"
Oct 30 23:42:32.289413 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:32.289386    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:42:32.296341 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:32.296313    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:42:33.630686 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:33.630650    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86dfbdff78-j4gh6"
Oct 30 23:42:34.042258 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:34.042215    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:34.045923 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:34.045893    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mxk2r"
Oct 30 23:42:35.850406 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:35.850378    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w2k95"
Oct 30 23:42:37.232973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:37.232940    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:38.135227 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:38.135201    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:38.135484 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:38.135391    6033 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:42:46.978620 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:46.978580    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:42:54.036099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:54.036061    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htb47"
Oct 30 23:42:55.267501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.267466    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:55.267906 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.267862    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="alertmanager" containerID="cri-o://edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc" gracePeriod=120
Oct 30 23:42:55.267973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.267919    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="prom-label-proxy" containerID="cri-o://f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736" gracePeriod=120
Oct 30 23:42:55.267973 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.267926    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-metric" containerID="cri-o://e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d" gracePeriod=120
Oct 30 23:42:55.268109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.268047    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-web" containerID="cri-o://df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0" gracePeriod=120
Oct 30 23:42:55.268109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.268069    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="config-reloader" containerID="cri-o://22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69" gracePeriod=120
Oct 30 23:42:55.268109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:55.268073    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy" containerID="cri-o://f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f" gracePeriod=120
Oct 30 23:42:56.086710 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086667    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736" exitCode=0
Oct 30 23:42:56.086710 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086703    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d" exitCode=0
Oct 30 23:42:56.086710 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086713    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f" exitCode=0
Oct 30 23:42:56.086710 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086723    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69" exitCode=0
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086731    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc" exitCode=0
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086734    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"}
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086822    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"}
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086863    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"}
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086900    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"}
Oct 30 23:42:56.087072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.086936    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"}
Oct 30 23:42:56.611282 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.611192    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748358    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748432    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748486    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748521    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748553    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748609    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748648    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748688    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748761    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748802    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748848    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748890    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.748926    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") pod \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\" (UID: \"e0784b5a-8ea2-47f2-898c-f8f3240a72b2\") "
Oct 30 23:42:56.758150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.755538    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:56.760046 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.759790    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:56.763255 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.763218    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:56.764178 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.764144    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:56.764931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.764902    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:56.766413 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.766385    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.766809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.766781    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.767234 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.767181    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.770413 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.770373    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "kube-api-access-9snn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:56.770535 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.770470    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.772133 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.772080    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.774443 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.774416    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.780378 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.780346    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config" pod "e0784b5a-8ea2-47f2-898c-f8f3240a72b2" (UID: "e0784b5a-8ea2-47f2-898c-f8f3240a72b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.849995 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.849963    6033 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-web-config\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.849995 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.849994    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9snn4\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-kube-api-access-9snn4\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.849995 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850005    6033 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-main-db\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850015    6033 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-cluster-tls-config\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850025    6033 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850048    6033 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850061    6033 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850070    6033 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-main-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850079    6033 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850089    6033 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850099    6033 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-config-out\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850108    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-metrics-client-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.850246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:56.850117    6033 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0784b5a-8ea2-47f2-898c-f8f3240a72b2-tls-assets\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:42:57.097958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.097239    6033 generic.go:356] "Generic (PLEG): container finished" podID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerID="df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0" exitCode=0
Oct 30 23:42:57.097958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.097316    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"}
Oct 30 23:42:57.097958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.097399    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e0784b5a-8ea2-47f2-898c-f8f3240a72b2","Type":"ContainerDied","Data":"dbf1401b61f4b1500665cb28f389426e9a69d5bf41765bb3a0cd4664a73ccb0c"}
Oct 30 23:42:57.097958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.097455    6033 scope.go:122] "RemoveContainer" containerID="f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"
Oct 30 23:42:57.097958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.097715    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.120809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.120783    6033 scope.go:122] "RemoveContainer" containerID="e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"
Oct 30 23:42:57.128933 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.128905    6033 scope.go:122] "RemoveContainer" containerID="f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"
Oct 30 23:42:57.137198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.137168    6033 scope.go:122] "RemoveContainer" containerID="df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"
Oct 30 23:42:57.145422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.145396    6033 scope.go:122] "RemoveContainer" containerID="22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"
Oct 30 23:42:57.154373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.154351    6033 scope.go:122] "RemoveContainer" containerID="edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"
Oct 30 23:42:57.188774 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.188746    6033 scope.go:122] "RemoveContainer" containerID="37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870"
Oct 30 23:42:57.196897 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.196872    6033 scope.go:122] "RemoveContainer" containerID="f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"
Oct 30 23:42:57.197261 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.197228    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736\": container with ID starting with f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736 not found: ID does not exist" containerID="f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"
Oct 30 23:42:57.197338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197276    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736"} err="failed to get container status \"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736\": rpc error: code = NotFound desc = could not find container \"f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736\": container with ID starting with f7036dd9d4ab14e9f6762502facfd21e3aadca627ce1bf33cd89da3ba544e736 not found: ID does not exist"
Oct 30 23:42:57.197338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197327    6033 scope.go:122] "RemoveContainer" containerID="e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"
Oct 30 23:42:57.197608 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.197590    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d\": container with ID starting with e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d not found: ID does not exist" containerID="e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"
Oct 30 23:42:57.197656 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197615    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d"} err="failed to get container status \"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d\": rpc error: code = NotFound desc = could not find container \"e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d\": container with ID starting with e51003f1afc289c21c091770611054493cecf63d49da88a33fdcf2729c9ae94d not found: ID does not exist"
Oct 30 23:42:57.197656 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197631    6033 scope.go:122] "RemoveContainer" containerID="f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"
Oct 30 23:42:57.197845 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.197823    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f\": container with ID starting with f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f not found: ID does not exist" containerID="f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"
Oct 30 23:42:57.197938 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197847    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f"} err="failed to get container status \"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f\": rpc error: code = NotFound desc = could not find container \"f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f\": container with ID starting with f5ebf647c89f066492b0a5ff0545417ca0874fa90eb6943586aa6e482705be4f not found: ID does not exist"
Oct 30 23:42:57.197938 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.197863    6033 scope.go:122] "RemoveContainer" containerID="df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"
Oct 30 23:42:57.198171 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.198150    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0\": container with ID starting with df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0 not found: ID does not exist" containerID="df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"
Oct 30 23:42:57.198219 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198182    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0"} err="failed to get container status \"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0\": rpc error: code = NotFound desc = could not find container \"df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0\": container with ID starting with df0d61906522471f2bfaead5f402eb89d54b0873edcf06c9707549a11035aec0 not found: ID does not exist"
Oct 30 23:42:57.198219 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198200    6033 scope.go:122] "RemoveContainer" containerID="22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"
Oct 30 23:42:57.198503 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.198486    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69\": container with ID starting with 22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69 not found: ID does not exist" containerID="22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"
Oct 30 23:42:57.198568 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198510    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69"} err="failed to get container status \"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69\": rpc error: code = NotFound desc = could not find container \"22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69\": container with ID starting with 22a448c06c048a56a6d07a3902e989fb806fdedde15f80380e25d23c3c0f7e69 not found: ID does not exist"
Oct 30 23:42:57.198568 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198529    6033 scope.go:122] "RemoveContainer" containerID="edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"
Oct 30 23:42:57.198786 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.198768    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc\": container with ID starting with edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc not found: ID does not exist" containerID="edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"
Oct 30 23:42:57.198852 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198791    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc"} err="failed to get container status \"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc\": rpc error: code = NotFound desc = could not find container \"edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc\": container with ID starting with edb6665dc89e4797c80bc701829bf3b3290531674000767b4afd592b175a18cc not found: ID does not exist"
Oct 30 23:42:57.198852 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.198808    6033 scope.go:122] "RemoveContainer" containerID="37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870"
Oct 30 23:42:57.199093 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:42:57.199074    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870\": container with ID starting with 37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870 not found: ID does not exist" containerID="37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870"
Oct 30 23:42:57.199152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.199100    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870"} err="failed to get container status \"37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870\": rpc error: code = NotFound desc = could not find container \"37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870\": container with ID starting with 37f288fca432116c8576e2a44604b63ec5633f6df783e198442c948f5bfc9870 not found: ID does not exist"
Oct 30 23:42:57.266122 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.266092    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:57.313486 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.313399    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:57.313921 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.313891    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:57.314240 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314225    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="prom-label-proxy"
Oct 30 23:42:57.314300 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314242    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="prom-label-proxy"
Oct 30 23:42:57.314300 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314257    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-web"
Oct 30 23:42:57.314300 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314263    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-web"
Oct 30 23:42:57.314300 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314282    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-metric"
Oct 30 23:42:57.314300 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314289    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-metric"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314301    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314308    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314333    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314355    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314362    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="alertmanager"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314367    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="alertmanager"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314376    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="init-config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314380    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="init-config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314433    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-metric"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314441    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314450    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314456    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="alertmanager"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314461    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="init-config-reloader"
Oct 30 23:42:57.314461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314466    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="prom-label-proxy"
Oct 30 23:42:57.314887 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.314473    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" containerName="kube-rbac-proxy-web"
Oct 30 23:42:57.338124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.337195    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435421    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435421    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435455    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435468    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435549    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7qdml\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435569    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435583    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435589    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435607    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.435609    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 23:42:57.436577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.436497    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:57.468272 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468239    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468432 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468289    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468432 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468353    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468432 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468374    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468432 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468397    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-volume\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468537    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-out\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468669 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468620    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468707 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468691    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468752 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468716    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-web-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468805 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468789    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468836    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468901 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468863    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.468937 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.468909    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdjf\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-kube-api-access-gfdjf\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570396    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570460    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570501    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-volume\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570566    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-out\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570597    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570627    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570654    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-web-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570686    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570723    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570751    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570783    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdjf\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-kube-api-access-gfdjf\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570815    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.570843    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.571630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.571351    6033 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:42:57.574734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.574708    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.575866 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.575835    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bef3cad5-0aa3-4b87-b3ed-74c60843a276-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.577275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.576482    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.582209 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.582154    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.582987    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.583296    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.583392    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-web-config\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.584478    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-out\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584688 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.584661    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584759 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.584733    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.584838 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.584821    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.585292 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.585273    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bef3cad5-0aa3-4b87-b3ed-74c60843a276-config-volume\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.638296 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.638259    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdjf\" (UniqueName: \"kubernetes.io/projected/bef3cad5-0aa3-4b87-b3ed-74c60843a276-kube-api-access-gfdjf\") pod \"alertmanager-main-0\" (UID: \"bef3cad5-0aa3-4b87-b3ed-74c60843a276\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.766515 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.766483    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 23:42:57.827481 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:57.827407    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0784b5a-8ea2-47f2-898c-f8f3240a72b2" path="/var/lib/kubelet/pods/e0784b5a-8ea2-47f2-898c-f8f3240a72b2/volumes"
Oct 30 23:42:58.265632 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:58.265580    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 23:42:58.267345 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:42:58.267309    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef3cad5_0aa3_4b87_b3ed_74c60843a276.slice/crio-4e8f0360662e9ebac3b45d23ae9df3f8db7a79242dab4763f12c5ea8e3046b32 WatchSource:0}: Error finding container 4e8f0360662e9ebac3b45d23ae9df3f8db7a79242dab4763f12c5ea8e3046b32: Status 404 returned error can't find the container with id 4e8f0360662e9ebac3b45d23ae9df3f8db7a79242dab4763f12c5ea8e3046b32
Oct 30 23:42:59.108199 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:59.108165    6033 generic.go:356] "Generic (PLEG): container finished" podID="bef3cad5-0aa3-4b87-b3ed-74c60843a276" containerID="b02cd71aeb2b18ae93f3f7e8826db05b9cdb2ddcf637dcb541568a074d6414d5" exitCode=0
Oct 30 23:42:59.108573 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:59.108205    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerDied","Data":"b02cd71aeb2b18ae93f3f7e8826db05b9cdb2ddcf637dcb541568a074d6414d5"}
Oct 30 23:42:59.108573 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:42:59.108257    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"4e8f0360662e9ebac3b45d23ae9df3f8db7a79242dab4763f12c5ea8e3046b32"}
Oct 30 23:43:00.116574 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116535    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"4475ab0ec8c26065ec3dd4a2d277bdea13fe5e7371800cbcb04fe265ef5aace0"}
Oct 30 23:43:00.117198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116626    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"9bd3e0d59d290301791234d1c014d08c7a5511c2a7a4f7e1ae2470d1a9277595"}
Oct 30 23:43:00.117198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116669    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"153c5162464512e2e15cc264e6cf193d6b6ff5a3b92f58798a8ab2d5eb9b8fdf"}
Oct 30 23:43:00.117198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116709    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"bad2c2e2a0c97657c08b7406e7dac0c1b24c6d8353cb15aaf0f79e39e1e4bf9b"}
Oct 30 23:43:00.117198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116751    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"2fdfbb881bfdcc1f0fd064077d435fbfaf95ecb94738bd46bc6e35992f1a9bbf"}
Oct 30 23:43:00.117198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.116787    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bef3cad5-0aa3-4b87-b3ed-74c60843a276","Type":"ContainerStarted","Data":"faea1a0bc325dbd1b405559a703f1071421644e2ec5e7272d0475b348b8a62bf"}
Oct 30 23:43:00.240290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:00.240236    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.233701733 podStartE2EDuration="3.240221611s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.519878ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:00.238920715 +0000 UTC m=+32.585904613" watchObservedRunningTime="2026-10-30 23:43:00.240221611 +0000 UTC m=+32.587205487"
Oct 30 23:43:01.092950 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.092916    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:01.093271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093246    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="prometheus" containerID="cri-o://dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba" gracePeriod=600
Oct 30 23:43:01.093359 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093272    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy" containerID="cri-o://4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e" gracePeriod=600
Oct 30 23:43:01.093359 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093330    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c" gracePeriod=600
Oct 30 23:43:01.093469 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093281    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="thanos-sidecar" containerID="cri-o://a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279" gracePeriod=600
Oct 30 23:43:01.093469 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093309    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-web" containerID="cri-o://f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d" gracePeriod=600
Oct 30 23:43:01.093589 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:01.093565    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="config-reloader" containerID="cri-o://0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390" gracePeriod=600
Oct 30 23:43:02.129501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129465    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c" exitCode=0
Oct 30 23:43:02.129501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129494    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e" exitCode=0
Oct 30 23:43:02.129501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129502    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279" exitCode=0
Oct 30 23:43:02.129501 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129508    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390" exitCode=0
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129519    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba" exitCode=0
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129547    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"}
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129610    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"}
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129637    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"}
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129663    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"}
Oct 30 23:43:02.129992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.129687    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"}
Oct 30 23:43:02.271183 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:02.271135    6033 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 23:43:02.271390 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:02.271205    6033 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 23:43:02.271390 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:02.271224    6033 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 23:43:02.271390 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.271240    6033 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 23:43:02.299239 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.299208    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dkcdb"
Oct 30 23:43:02.378184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.378149    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:02.521276 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521240    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521276 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521282    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521315    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521337    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521362    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521401    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521432    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521475    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521513    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521573    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521605    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521641    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521686    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521726    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521755    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521771    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521803    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521839    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521878    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.521988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.521966    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522007    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522079    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") pod \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\" (UID: \"5200d5bb-5437-456e-93bb-3c95d0e19d1d\") "
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522081    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522178    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522477    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522481    6033 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522510    6033 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522526    6033 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.522544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522542    6033 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.522977 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.522762    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.524175 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.523959    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.524777 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.524738    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.525273 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.525234    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "kube-api-access-lz8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:02.526716 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.526576    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.526716 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.526697    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.526898 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.526865    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527003 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.526976    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527145 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527127    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527243 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527215    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527328 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527272    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527417 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527392    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.527475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527433    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:02.527475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.527460    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.536709 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.536677    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config" pod "5200d5bb-5437-456e-93bb-3c95d0e19d1d" (UID: "5200d5bb-5437-456e-93bb-3c95d0e19d1d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:02.623097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623058    6033 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-metrics-client-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623092    6033 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config-out\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623102    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8wn\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-kube-api-access-lz8wn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623113    6033 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-web-config\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623122    6033 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-metrics-client-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623135    6033 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623143    6033 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5200d5bb-5437-456e-93bb-3c95d0e19d1d-tls-assets\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623153    6033 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623162    6033 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623171    6033 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623180    6033 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-config\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623188    6033 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-kube-rbac-proxy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623198    6033 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-grpc-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623207    6033 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5200d5bb-5437-456e-93bb-3c95d0e19d1d-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623217    6033 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.623329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:02.623227    6033 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5200d5bb-5437-456e-93bb-3c95d0e19d1d-prometheus-k8s-db\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:03.136386 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.136352    6033 generic.go:356] "Generic (PLEG): container finished" podID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerID="f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d" exitCode=0
Oct 30 23:43:03.136831 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.136416    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"}
Oct 30 23:43:03.136831 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.136443    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.136831 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.136478    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5200d5bb-5437-456e-93bb-3c95d0e19d1d","Type":"ContainerDied","Data":"8f4e727345b2de12e12defd3e5be259dfe06fece1c4a7cb2783664a2d4d4427f"}
Oct 30 23:43:03.136831 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.136512    6033 scope.go:122] "RemoveContainer" containerID="d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"
Oct 30 23:43:03.146598 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.146569    6033 scope.go:122] "RemoveContainer" containerID="4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"
Oct 30 23:43:03.154934 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.154898    6033 scope.go:122] "RemoveContainer" containerID="f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"
Oct 30 23:43:03.165200 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.165165    6033 scope.go:122] "RemoveContainer" containerID="a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"
Oct 30 23:43:03.178138 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.178108    6033 scope.go:122] "RemoveContainer" containerID="0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"
Oct 30 23:43:03.186314 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.186289    6033 scope.go:122] "RemoveContainer" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"
Oct 30 23:43:03.194896 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.194864    6033 scope.go:122] "RemoveContainer" containerID="c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd"
Oct 30 23:43:03.202913 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.202891    6033 scope.go:122] "RemoveContainer" containerID="d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"
Oct 30 23:43:03.203278 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.203258    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c\": container with ID starting with d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c not found: ID does not exist" containerID="d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"
Oct 30 23:43:03.203327 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203295    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c"} err="failed to get container status \"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c\": rpc error: code = NotFound desc = could not find container \"d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c\": container with ID starting with d13de724b905d250a1c60b81093b43d6cb51c908529d1092918acc1e8ca5184c not found: ID does not exist"
Oct 30 23:43:03.203327 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203314    6033 scope.go:122] "RemoveContainer" containerID="4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"
Oct 30 23:43:03.203580 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.203561    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e\": container with ID starting with 4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e not found: ID does not exist" containerID="4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"
Oct 30 23:43:03.203626 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203587    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e"} err="failed to get container status \"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e\": rpc error: code = NotFound desc = could not find container \"4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e\": container with ID starting with 4ac6a3a64d146bb00472af78b334d23c7145e5a6830c45a54e413e32dd8d333e not found: ID does not exist"
Oct 30 23:43:03.203626 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203605    6033 scope.go:122] "RemoveContainer" containerID="f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"
Oct 30 23:43:03.203811 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.203796    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d\": container with ID starting with f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d not found: ID does not exist" containerID="f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"
Oct 30 23:43:03.203853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203814    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d"} err="failed to get container status \"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d\": rpc error: code = NotFound desc = could not find container \"f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d\": container with ID starting with f06305f252009aeb70916ab7a5ca19f635073b19243f680c4e986df9893e7c2d not found: ID does not exist"
Oct 30 23:43:03.203853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.203826    6033 scope.go:122] "RemoveContainer" containerID="a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"
Oct 30 23:43:03.204081 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.204060    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279\": container with ID starting with a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279 not found: ID does not exist" containerID="a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"
Oct 30 23:43:03.204153 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204090    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279"} err="failed to get container status \"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279\": rpc error: code = NotFound desc = could not find container \"a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279\": container with ID starting with a04c0b32af4f4b0f04592bd6eb63de07d526a28032198ad5b755094833ca3279 not found: ID does not exist"
Oct 30 23:43:03.204153 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204110    6033 scope.go:122] "RemoveContainer" containerID="0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"
Oct 30 23:43:03.204388 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.204374    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390\": container with ID starting with 0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390 not found: ID does not exist" containerID="0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"
Oct 30 23:43:03.204427 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204390    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390"} err="failed to get container status \"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390\": rpc error: code = NotFound desc = could not find container \"0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390\": container with ID starting with 0085fc743a322c12459b4ea55235c04f8700de22286646470875948e3731c390 not found: ID does not exist"
Oct 30 23:43:03.204427 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204402    6033 scope.go:122] "RemoveContainer" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"
Oct 30 23:43:03.204625 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.204609    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba\": container with ID starting with dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba not found: ID does not exist" containerID="dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"
Oct 30 23:43:03.204671 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204630    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba"} err="failed to get container status \"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba\": rpc error: code = NotFound desc = could not find container \"dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba\": container with ID starting with dcc42257c43230b4ee8c0f2269ee69557a2684e6d9254e4e91027d7ccd898bba not found: ID does not exist"
Oct 30 23:43:03.204671 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204641    6033 scope.go:122] "RemoveContainer" containerID="c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd"
Oct 30 23:43:03.204834 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:03.204816    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd\": container with ID starting with c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd not found: ID does not exist" containerID="c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd"
Oct 30 23:43:03.204879 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.204839    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd"} err="failed to get container status \"c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd\": rpc error: code = NotFound desc = could not find container \"c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd\": container with ID starting with c4836342a4b9a9a62acb2cbec3e196fe16f54fbab1a1dd2731e557add0f549cd not found: ID does not exist"
Oct 30 23:43:03.340400 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.340368    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:03.387256 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387165    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:03.387581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387557    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:03.387872 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387859    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="prometheus"
Oct 30 23:43:03.387908 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387874    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="prometheus"
Oct 30 23:43:03.387908 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387885    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="init-config-reloader"
Oct 30 23:43:03.387908 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387890    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="init-config-reloader"
Oct 30 23:43:03.387908 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387901    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="config-reloader"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387909    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="config-reloader"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387934    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-thanos"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387939    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-thanos"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387951    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387956    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387962    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-web"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387967    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-web"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387976    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="thanos-sidecar"
Oct 30 23:43:03.388048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.387982    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="thanos-sidecar"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388070    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388080    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-web"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388087    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="thanos-sidecar"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388094    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="prometheus"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388099    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="init-config-reloader"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388105    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="config-reloader"
Oct 30 23:43:03.388312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.388114    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" containerName="kube-rbac-proxy-thanos"
Oct 30 23:43:03.420522 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.420489    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.470222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470176    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 23:43:03.470384 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470234    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 23:43:03.470565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470531    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 23:43:03.470565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470544    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 23:43:03.470565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470545    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470581    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470639    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470663    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-qbwc4\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470679    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470711    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470755    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8rm397jpo960s\""
Oct 30 23:43:03.470814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470798    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 23:43:03.471072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470907    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 23:43:03.471072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470912    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 23:43:03.471072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.470914    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 23:43:03.517573 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.517543    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 23:43:03.517833 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.517815    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 23:43:03.531258 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531228    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531266    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531287    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531307    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531445 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531427    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531477    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531553 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531538    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531584 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531564    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531618 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531582    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.531970 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.531777    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532123 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532098    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config-out\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532188 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532170    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-web-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532252 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532216    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532256    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532352 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532298    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532352 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532324    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532373    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmfd\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-kube-api-access-pbmfd\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532404    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532535 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532454    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.532535 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.532487    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633089    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633133    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633159    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633199    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmfd\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-kube-api-access-pbmfd\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633222    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633252    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633277    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633303    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633327    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633374 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633356    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633390    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633425    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633491    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633521    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633590    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633599    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633619    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633640    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633644    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.633700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633700    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config-out\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.634160 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633731    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-web-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.634160 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.633756    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.634592 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.634564    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.635137 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.635116    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.636212 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.636187    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.637054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.636241    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.637308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.637260    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.638329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.637650    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config-out\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.638766 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.638740    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.639409 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.638973    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.639409 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.639364    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.640312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.640293    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.641092 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.641065    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/47de0f8d-fff4-4d27-bc35-bc926d39e08d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.641318 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.641290    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.642136 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.642117    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.642376 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.642351    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.642913 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.642893    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.643097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.643073    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.643578 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.643561    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-web-config\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.644449 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.644423    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/47de0f8d-fff4-4d27-bc35-bc926d39e08d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.693625 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.693588    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmfd\" (UniqueName: \"kubernetes.io/projected/47de0f8d-fff4-4d27-bc35-bc926d39e08d-kube-api-access-pbmfd\") pod \"prometheus-k8s-0\" (UID: \"47de0f8d-fff4-4d27-bc35-bc926d39e08d\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:03.826565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.826527    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5200d5bb-5437-456e-93bb-3c95d0e19d1d" path="/var/lib/kubelet/pods/5200d5bb-5437-456e-93bb-3c95d0e19d1d/volumes"
Oct 30 23:43:03.882954 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:03.882910    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:04.385624 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:04.385588    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 23:43:04.388425 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:43:04.388388    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47de0f8d_fff4_4d27_bc35_bc926d39e08d.slice/crio-14d88836287f95481db06bd830ee921a76fe601fe48d0df141e1ccdd71ef8c7a WatchSource:0}: Error finding container 14d88836287f95481db06bd830ee921a76fe601fe48d0df141e1ccdd71ef8c7a: Status 404 returned error can't find the container with id 14d88836287f95481db06bd830ee921a76fe601fe48d0df141e1ccdd71ef8c7a
Oct 30 23:43:05.146423 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:05.146387    6033 generic.go:356] "Generic (PLEG): container finished" podID="47de0f8d-fff4-4d27-bc35-bc926d39e08d" containerID="7cf146681ab1af17d4baa171da40ed0c5436a6f6e259d1aaab182c3b33fd28aa" exitCode=0
Oct 30 23:43:05.146582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:05.146460    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerDied","Data":"7cf146681ab1af17d4baa171da40ed0c5436a6f6e259d1aaab182c3b33fd28aa"}
Oct 30 23:43:05.146582 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:05.146518    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"14d88836287f95481db06bd830ee921a76fe601fe48d0df141e1ccdd71ef8c7a"}
Oct 30 23:43:06.153478 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153440    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"8ae537cd1526aec950297b8fc50fef27ab7b0f0f0dfeeff12f68761d0ac06bca"}
Oct 30 23:43:06.153924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153507    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"64ce9f553710f92167c8ac6e1c3880ef6d6caed7c871024dda6acf3fa29dfb4a"}
Oct 30 23:43:06.153924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153540    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"dafe7c534ae64ec5aeff4c1d00bb120c2b0b58b42812b81811fda36cbe5e67b9"}
Oct 30 23:43:06.153924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153564    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"2e9e278f3c61dbe6359fd72a47aaa765f5fb39c6f7337254b941c90c5ddedb60"}
Oct 30 23:43:06.153924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153588    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"a6ba54d2b0438595dde080bf26864c3ec25db0b6705febb815d1889869b691fd"}
Oct 30 23:43:06.153924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.153611    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"47de0f8d-fff4-4d27-bc35-bc926d39e08d","Type":"ContainerStarted","Data":"2c2bd89b7f66a36d6753a53642ae4dc671b1e9eec396a745d8fa69b58f505b68"}
Oct 30 23:43:06.280863 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:06.280804    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.26125936 podStartE2EDuration="3.280786859s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.527499ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:06.27979852 +0000 UTC m=+38.626782418" watchObservedRunningTime="2026-10-30 23:43:06.280786859 +0000 UTC m=+38.627770752"
Oct 30 23:43:08.883679 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:08.883635    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:16.968503 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:16.968459    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:43:21.180791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:21.180730    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:21.197254 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:21.197215    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:21.232650 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:43:21.232623    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 23:43:46.967099 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:43:46.967065    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:44:16.967860 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:44:16.967777    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:44:46.968828 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:44:46.968785    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:45:16.969080 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:45:16.969017    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:45:46.968968 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:45:46.968882    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:45:53.929564 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:53.929525    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"]
Oct 30 23:45:53.932220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:53.932198    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:53.983613 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:53.983577    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 23:45:53.983613 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:53.983600    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 23:45:53.983613 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:53.983596    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pxfm6\""
Oct 30 23:45:54.029733 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.029698    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 23:45:54.029733 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.029703    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 23:45:54.029733 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.029703    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 23:45:54.030914 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.030893    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"]
Oct 30 23:45:54.092841 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.092805    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.093024 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.092860    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.093024 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.092941    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.093024 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.092977    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.093153 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.093077    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26v6s\" (UniqueName: \"kubernetes.io/projected/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-kube-api-access-26v6s\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.093153 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.093123    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194071 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.193969    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194071 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194047    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194071 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194071    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194357 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194086    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194357 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194109    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26v6s\" (UniqueName: \"kubernetes.io/projected/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-kube-api-access-26v6s\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194357 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194136    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.194562 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.194539    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-manifests\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.196575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.196546    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-tmp\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.196796 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.196776    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.196900 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.196880    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.196957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.196881    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.255918 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.255887    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26v6s\" (UniqueName: \"kubernetes.io/projected/d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60-kube-api-access-26v6s\") pod \"opendatahub-operator-controller-manager-6bfcd988bb-h9d2s\" (UID: \"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:54.545906 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:54.545867    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:45:55.080121 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.080092    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"]
Oct 30 23:45:55.080474 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:45:55.080126    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67d7eb4_fcda_4cf0_9a9b_84a9e8fe0f60.slice/crio-e7849f71bee609bd6aa886baa44a6c3ee24cb338c37209f2f6fb833901be4fd8 WatchSource:0}: Error finding container e7849f71bee609bd6aa886baa44a6c3ee24cb338c37209f2f6fb833901be4fd8: Status 404 returned error can't find the container with id e7849f71bee609bd6aa886baa44a6c3ee24cb338c37209f2f6fb833901be4fd8
Oct 30 23:45:55.081749 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.081730    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:45:55.410931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.410840    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"]
Oct 30 23:45:55.415068 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.415045    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.467601 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.467570    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 23:45:55.467901 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.467885    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 23:45:55.467983 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.467907    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2vmhq\""
Oct 30 23:45:55.474459 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.474429    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"]
Oct 30 23:45:55.606859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.606820    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.606859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.606864    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrlc\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-kube-api-access-xmrlc\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.707848 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.707751    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.707848 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.707794    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrlc\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-kube-api-access-xmrlc\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.767748 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.767713    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.767988 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.767958    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrlc\" (UniqueName: \"kubernetes.io/projected/ebacd3ee-1232-4e08-8ebb-e19b8c3ba861-kube-api-access-xmrlc\") pod \"cert-manager-webhook-5dbbbf75cc-vvwt4\" (UID: \"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:55.783849 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:55.783813    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s" event={"ID":"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60","Type":"ContainerStarted","Data":"e7849f71bee609bd6aa886baa44a6c3ee24cb338c37209f2f6fb833901be4fd8"}
Oct 30 23:45:56.042965 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:56.042928    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:45:56.413492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:56.413389    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"]
Oct 30 23:45:56.418182 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:45:56.418148    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebacd3ee_1232_4e08_8ebb_e19b8c3ba861.slice/crio-34693ed7c68c6ca63ad893073fc2d5154baeab7dd33ebd2dfcb94519e9ffc8bb WatchSource:0}: Error finding container 34693ed7c68c6ca63ad893073fc2d5154baeab7dd33ebd2dfcb94519e9ffc8bb: Status 404 returned error can't find the container with id 34693ed7c68c6ca63ad893073fc2d5154baeab7dd33ebd2dfcb94519e9ffc8bb
Oct 30 23:45:56.790492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:56.790446    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4" event={"ID":"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861","Type":"ContainerStarted","Data":"34693ed7c68c6ca63ad893073fc2d5154baeab7dd33ebd2dfcb94519e9ffc8bb"}
Oct 30 23:45:59.806635 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:59.806599    6033 generic.go:356] "Generic (PLEG): container finished" podID="d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60" containerID="bc4120a46074e1ca774dd7fed94b0c42c6272be5a875a24225ebd9314cc0072e" exitCode=0
Oct 30 23:45:59.807196 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:45:59.806690    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s" event={"ID":"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60","Type":"ContainerDied","Data":"bc4120a46074e1ca774dd7fed94b0c42c6272be5a875a24225ebd9314cc0072e"}
Oct 30 23:46:00.813106 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:00.813067    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s" event={"ID":"d67d7eb4-fcda-4cf0-9a9b-84a9e8fe0f60","Type":"ContainerStarted","Data":"8737f779e3e957015637b47cbce63dcde27c143ae897ed331e6c749ba049bd47"}
Oct 30 23:46:00.813534 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:00.813362    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:46:01.061958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:01.061884    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s" podStartSLOduration=2.767653005 podStartE2EDuration="8.061865668s" totalImagesPullingTime="4.639555332s" totalInitContainerRuntime="654.657331ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:01.015199219 +0000 UTC m=+213.362183120" watchObservedRunningTime="2026-10-30 23:46:01.061865668 +0000 UTC m=+213.408849563"
Oct 30 23:46:01.818605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:01.818569    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4" event={"ID":"ebacd3ee-1232-4e08-8ebb-e19b8c3ba861","Type":"ContainerStarted","Data":"dd8bc8ab36c49a395632f26d40cfa56479225813ae213ee25d621553fe95ac22"}
Oct 30 23:46:01.819114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:01.818727    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:46:07.826773 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:07.826748    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4"
Oct 30 23:46:07.940704 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:07.940648    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vvwt4" podStartSLOduration=8.275635686 podStartE2EDuration="12.94063388s" totalImagesPullingTime="4.664998194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:01.940801334 +0000 UTC m=+214.287785233" watchObservedRunningTime="2026-10-30 23:46:07.94063388 +0000 UTC m=+220.287617756"
Oct 30 23:46:08.249099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.248985    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"]
Oct 30 23:46:08.252532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.252501    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.308366 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.308331    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:46:08.309428 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.309399    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"]
Oct 30 23:46:08.316636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.316608    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhghj\" (UniqueName: \"kubernetes.io/projected/6ad44b39-6226-4c5b-bda7-99907d7f9b18-kube-api-access-xhghj\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.316774 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.316666    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad44b39-6226-4c5b-bda7-99907d7f9b18-tmp\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.355194 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.355157    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 23:46:08.355383 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.355217    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-mmz2z\""
Oct 30 23:46:08.417694 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.417659    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhghj\" (UniqueName: \"kubernetes.io/projected/6ad44b39-6226-4c5b-bda7-99907d7f9b18-kube-api-access-xhghj\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.417907 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.417722    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad44b39-6226-4c5b-bda7-99907d7f9b18-tmp\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.418171 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.418150    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ad44b39-6226-4c5b-bda7-99907d7f9b18-tmp\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.475938 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.475900    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhghj\" (UniqueName: \"kubernetes.io/projected/6ad44b39-6226-4c5b-bda7-99907d7f9b18-kube-api-access-xhghj\") pod \"jobset-operator-674cd59dc5-lbmjl\" (UID: \"6ad44b39-6226-4c5b-bda7-99907d7f9b18\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.564040 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.564007    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"
Oct 30 23:46:08.935865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:08.935787    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl"]
Oct 30 23:46:08.939220 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:08.939188    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ad44b39_6226_4c5b_bda7_99907d7f9b18.slice/crio-c890e4d2850a8eff6bce8fd8bc9b821175ccbcb48593bce8c9c1b7004f244287 WatchSource:0}: Error finding container c890e4d2850a8eff6bce8fd8bc9b821175ccbcb48593bce8c9c1b7004f244287: Status 404 returned error can't find the container with id c890e4d2850a8eff6bce8fd8bc9b821175ccbcb48593bce8c9c1b7004f244287
Oct 30 23:46:09.853220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:09.853185    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl" event={"ID":"6ad44b39-6226-4c5b-bda7-99907d7f9b18","Type":"ContainerStarted","Data":"c890e4d2850a8eff6bce8fd8bc9b821175ccbcb48593bce8c9c1b7004f244287"}
Oct 30 23:46:11.827379 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:11.827347    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bfcd988bb-h9d2s"
Oct 30 23:46:13.873464 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:13.873374    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl" event={"ID":"6ad44b39-6226-4c5b-bda7-99907d7f9b18","Type":"ContainerStarted","Data":"87203aedbe8ae01bb560188ec0202ccddd532ac36ff15d47faf3870329123e91"}
Oct 30 23:46:13.994077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:13.994002    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-lbmjl" podStartSLOduration=1.419429179 podStartE2EDuration="5.9939858s" totalImagesPullingTime="4.574556621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:13.993551036 +0000 UTC m=+226.340534934" watchObservedRunningTime="2026-10-30 23:46:13.9939858 +0000 UTC m=+226.340969697"
Oct 30 23:46:15.345874 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.345839    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-smpqz"]
Oct 30 23:46:15.348080 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.348062    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.382888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.382854    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwl5\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-kube-api-access-hqwl5\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.383089 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.382980    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-bound-sa-token\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.409000 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.408969    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-smpqz"]
Oct 30 23:46:15.483889 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.483850    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-bound-sa-token\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.484099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.483910    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwl5\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-kube-api-access-hqwl5\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.489873 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.489854    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-b6xs6\""
Oct 30 23:46:15.541188 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.541153    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-bound-sa-token\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.542242 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.542210    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwl5\" (UniqueName: \"kubernetes.io/projected/49ec94dc-0573-4c4b-82f9-6240f433b289-kube-api-access-hqwl5\") pod \"cert-manager-597b95cb84-smpqz\" (UID: \"49ec94dc-0573-4c4b-82f9-6240f433b289\") " pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:15.658771 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:15.658695    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-smpqz"
Oct 30 23:46:16.030804 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:16.030774    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-smpqz"]
Oct 30 23:46:16.033829 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:16.033688    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49ec94dc_0573_4c4b_82f9_6240f433b289.slice/crio-8bfb8209d3d7f7d2b69d389a88d888cad78444dabc18b1aa05ac62919995585a WatchSource:0}: Error finding container 8bfb8209d3d7f7d2b69d389a88d888cad78444dabc18b1aa05ac62919995585a: Status 404 returned error can't find the container with id 8bfb8209d3d7f7d2b69d389a88d888cad78444dabc18b1aa05ac62919995585a
Oct 30 23:46:16.886748 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:16.886712    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-smpqz" event={"ID":"49ec94dc-0573-4c4b-82f9-6240f433b289","Type":"ContainerStarted","Data":"56715c524615a64e20abdae9434eadbd7bb239d12ba6314dd3f03d19bb3c6975"}
Oct 30 23:46:16.887255 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:16.886758    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-smpqz" event={"ID":"49ec94dc-0573-4c4b-82f9-6240f433b289","Type":"ContainerStarted","Data":"8bfb8209d3d7f7d2b69d389a88d888cad78444dabc18b1aa05ac62919995585a"}
Oct 30 23:46:16.969634 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:46:16.969580    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:46:17.016168 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:17.016115    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-smpqz" podStartSLOduration=2.01610094 podStartE2EDuration="2.01610094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:17.014112545 +0000 UTC m=+229.361096443" watchObservedRunningTime="2026-10-30 23:46:17.01610094 +0000 UTC m=+229.363084838"
Oct 30 23:46:24.351341 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.351304    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"]
Oct 30 23:46:24.354199 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.354173    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.404329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.404289    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 23:46:24.404329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.404302    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 23:46:24.404329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.404302    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 23:46:24.451208 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.451168    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 23:46:24.451412 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.451243    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rbwfv\""
Oct 30 23:46:24.452103 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.452081    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"]
Oct 30 23:46:24.467610 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.467577    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sstj\" (UniqueName: \"kubernetes.io/projected/442fa4e9-64b3-48ce-ac9f-3bec3e289328-kube-api-access-8sstj\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.467714 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.467627    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/442fa4e9-64b3-48ce-ac9f-3bec3e289328-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.467714 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.467655    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/442fa4e9-64b3-48ce-ac9f-3bec3e289328-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.569114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.569066    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sstj\" (UniqueName: \"kubernetes.io/projected/442fa4e9-64b3-48ce-ac9f-3bec3e289328-kube-api-access-8sstj\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.569114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.569118    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/442fa4e9-64b3-48ce-ac9f-3bec3e289328-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.569322 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.569227    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/442fa4e9-64b3-48ce-ac9f-3bec3e289328-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.569889 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.569870    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/442fa4e9-64b3-48ce-ac9f-3bec3e289328-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.571590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.571568    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/442fa4e9-64b3-48ce-ac9f-3bec3e289328-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.647625 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.647540    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sstj\" (UniqueName: \"kubernetes.io/projected/442fa4e9-64b3-48ce-ac9f-3bec3e289328-kube-api-access-8sstj\") pod \"kuadrant-console-plugin-66c4fb575d-8k6m8\" (UID: \"442fa4e9-64b3-48ce-ac9f-3bec3e289328\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:24.665222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:24.665183    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"
Oct 30 23:46:25.027525 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:25.027498    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8"]
Oct 30 23:46:25.030291 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:25.030261    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod442fa4e9_64b3_48ce_ac9f_3bec3e289328.slice/crio-419caaf2599a81eaa8c254b25e933b5818830fda5d40e9e7735d87f35536ecda WatchSource:0}: Error finding container 419caaf2599a81eaa8c254b25e933b5818830fda5d40e9e7735d87f35536ecda: Status 404 returned error can't find the container with id 419caaf2599a81eaa8c254b25e933b5818830fda5d40e9e7735d87f35536ecda
Oct 30 23:46:25.924261 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:25.924224    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8" event={"ID":"442fa4e9-64b3-48ce-ac9f-3bec3e289328","Type":"ContainerStarted","Data":"419caaf2599a81eaa8c254b25e933b5818830fda5d40e9e7735d87f35536ecda"}
Oct 30 23:46:29.884565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:29.884526    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"]
Oct 30 23:46:29.901129 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:29.901097    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:29.957371 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:29.957341    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:46:29.957546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:29.957409    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:46:29.964372 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:29.964292    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"]
Oct 30 23:46:30.022332 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.022289    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q8vk7\""
Oct 30 23:46:30.022332 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.022306    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 23:46:30.022332 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.022314    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 23:46:30.023809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.023780    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczvg\" (UniqueName: \"kubernetes.io/projected/ac2b7908-97d9-4867-adef-411bfd62188a-kube-api-access-jczvg\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.023949 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.023826    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2b7908-97d9-4867-adef-411bfd62188a-tmp\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.023949 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.023887    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2b7908-97d9-4867-adef-411bfd62188a-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.124534 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.124498    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2b7908-97d9-4867-adef-411bfd62188a-tmp\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.124737 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.124589    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2b7908-97d9-4867-adef-411bfd62188a-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.124737 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.124665    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczvg\" (UniqueName: \"kubernetes.io/projected/ac2b7908-97d9-4867-adef-411bfd62188a-kube-api-access-jczvg\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.127234 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.127201    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac2b7908-97d9-4867-adef-411bfd62188a-tmp\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.127545 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.127514    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac2b7908-97d9-4867-adef-411bfd62188a-tls-certs\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.183255 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.183174    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczvg\" (UniqueName: \"kubernetes.io/projected/ac2b7908-97d9-4867-adef-411bfd62188a-kube-api-access-jczvg\") pod \"kube-auth-proxy-546bf7cb95-r7gvf\" (UID: \"ac2b7908-97d9-4867-adef-411bfd62188a\") " pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.216989 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.216950    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"
Oct 30 23:46:30.593664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.593631    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf"]
Oct 30 23:46:30.597384 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:30.597342    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2b7908_97d9_4867_adef_411bfd62188a.slice/crio-4b0a6182e7ad1676855b5ffcde48975b446cd22fb373ea98cef1953f5e7deafe WatchSource:0}: Error finding container 4b0a6182e7ad1676855b5ffcde48975b446cd22fb373ea98cef1953f5e7deafe: Status 404 returned error can't find the container with id 4b0a6182e7ad1676855b5ffcde48975b446cd22fb373ea98cef1953f5e7deafe
Oct 30 23:46:30.949156 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:30.949064    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf" event={"ID":"ac2b7908-97d9-4867-adef-411bfd62188a","Type":"ContainerStarted","Data":"4b0a6182e7ad1676855b5ffcde48975b446cd22fb373ea98cef1953f5e7deafe"}
Oct 30 23:46:31.955175 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:31.955138    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8" event={"ID":"442fa4e9-64b3-48ce-ac9f-3bec3e289328","Type":"ContainerStarted","Data":"160ff7d688d4f7d3490e4cc61a316b6ed954c1e5bd5037fa7e50d58769f130b2"}
Oct 30 23:46:32.084814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:32.084763    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-8k6m8" podStartSLOduration=1.7730390310000002 podStartE2EDuration="8.084746349s" totalImagesPullingTime="6.311707318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:32.084599503 +0000 UTC m=+244.431583401" watchObservedRunningTime="2026-10-30 23:46:32.084746349 +0000 UTC m=+244.431730243"
Oct 30 23:46:33.163585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.163540    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:33.169786 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.169749    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.222677 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.222609    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 23:46:33.223660 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.223628    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:33.254147 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.254115    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.254147 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.254158    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsf2w\" (UniqueName: \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.355420 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.355382    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.355600 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.355441    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsf2w\" (UniqueName: \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.356114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.356089    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.414709 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.414632    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:33.423507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.423475    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsf2w\" (UniqueName: \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\") pod \"limitador-limitador-74dfcfb894-r4kdw\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.483985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.483955    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:33.913393 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.913361    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:33.917237 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:33.917195    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafa7493_b13b_4a1f_b0a0_d7c85ddf80a0.slice/crio-c2660abe094c416e22aa487c62c0d430f2eb666bf05dacb8cc77c0ac7c09e6af WatchSource:0}: Error finding container c2660abe094c416e22aa487c62c0d430f2eb666bf05dacb8cc77c0ac7c09e6af: Status 404 returned error can't find the container with id c2660abe094c416e22aa487c62c0d430f2eb666bf05dacb8cc77c0ac7c09e6af
Oct 30 23:46:33.966513 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:33.966473    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" event={"ID":"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0","Type":"ContainerStarted","Data":"c2660abe094c416e22aa487c62c0d430f2eb666bf05dacb8cc77c0ac7c09e6af"}
Oct 30 23:46:34.971969 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:34.971926    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf" event={"ID":"ac2b7908-97d9-4867-adef-411bfd62188a","Type":"ContainerStarted","Data":"0015525f84b2fc03e1b85bd65d34a9485151865beb35151200ddd0748ab3fc69"}
Oct 30 23:46:35.082549 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:35.082497    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546bf7cb95-r7gvf" podStartSLOduration=2.516604358 podStartE2EDuration="6.082481277s" totalImagesPullingTime="3.565876919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:35.081120649 +0000 UTC m=+247.428104547" watchObservedRunningTime="2026-10-30 23:46:35.082481277 +0000 UTC m=+247.429465184"
Oct 30 23:46:36.299890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.299855    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:46:36.303506 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.303478    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:46:36.306209 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.306181    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:46:36.309228 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.309208    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.354025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.353986    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dn9sl\""
Oct 30 23:46:36.355352 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.355328    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:46:36.408523 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.408479    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:46:36.484157 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.484123    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk89r\" (UniqueName: \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\") pod \"authorino-56c6c9ddd5-xfdks\" (UID: \"2223dd19-7764-474b-94bb-b8e0d9c94d4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:46:36.484330 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.484188    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442mx\" (UniqueName: \"kubernetes.io/projected/34e7bbb9-29be-4332-8ddd-34b2e683bd28-kube-api-access-442mx\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.484330 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.484285    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/34e7bbb9-29be-4332-8ddd-34b2e683bd28-config-file\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.585154 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.585066    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/34e7bbb9-29be-4332-8ddd-34b2e683bd28-config-file\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.585312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.585161    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk89r\" (UniqueName: \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\") pod \"authorino-56c6c9ddd5-xfdks\" (UID: \"2223dd19-7764-474b-94bb-b8e0d9c94d4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:46:36.585312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.585232    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442mx\" (UniqueName: \"kubernetes.io/projected/34e7bbb9-29be-4332-8ddd-34b2e683bd28-kube-api-access-442mx\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.585813 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.585793    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/34e7bbb9-29be-4332-8ddd-34b2e683bd28-config-file\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.670465 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.670426    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:46:36.671549 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.671519    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk89r\" (UniqueName: \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\") pod \"authorino-56c6c9ddd5-xfdks\" (UID: \"2223dd19-7764-474b-94bb-b8e0d9c94d4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:46:36.673557 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.673515    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442mx\" (UniqueName: \"kubernetes.io/projected/34e7bbb9-29be-4332-8ddd-34b2e683bd28-kube-api-access-442mx\") pod \"limitador-limitador-7c9b87d574-4n4cm\" (UID: \"34e7bbb9-29be-4332-8ddd-34b2e683bd28\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:36.921250 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.921167    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:46:36.929557 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:36.929524    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:37.323568 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.323539    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:46:37.326166 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:37.326133    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e7bbb9_29be_4332_8ddd_34b2e683bd28.slice/crio-7d7862d578dbccef491539966b0ec99577b996e265bc32f1b42fa17df6a101d0 WatchSource:0}: Error finding container 7d7862d578dbccef491539966b0ec99577b996e265bc32f1b42fa17df6a101d0: Status 404 returned error can't find the container with id 7d7862d578dbccef491539966b0ec99577b996e265bc32f1b42fa17df6a101d0
Oct 30 23:46:37.368454 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.368422    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:46:37.371966 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:46:37.371930    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2223dd19_7764_474b_94bb_b8e0d9c94d4c.slice/crio-d082ff3a51cfb21f79e021328dd766d68e2c8593f50cf77bfb422d2520e4496d WatchSource:0}: Error finding container d082ff3a51cfb21f79e021328dd766d68e2c8593f50cf77bfb422d2520e4496d: Status 404 returned error can't find the container with id d082ff3a51cfb21f79e021328dd766d68e2c8593f50cf77bfb422d2520e4496d
Oct 30 23:46:37.990182 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.990140    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" event={"ID":"2223dd19-7764-474b-94bb-b8e0d9c94d4c","Type":"ContainerStarted","Data":"d082ff3a51cfb21f79e021328dd766d68e2c8593f50cf77bfb422d2520e4496d"}
Oct 30 23:46:37.991683 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.991654    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" event={"ID":"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0","Type":"ContainerStarted","Data":"d81aba5d0ac694b6bbb707f6711cb8a362e97a0c50d3cf1540061b8bf04fcae0"}
Oct 30 23:46:37.991860 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.991776    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:37.992941 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:37.992920    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm" event={"ID":"34e7bbb9-29be-4332-8ddd-34b2e683bd28","Type":"ContainerStarted","Data":"7d7862d578dbccef491539966b0ec99577b996e265bc32f1b42fa17df6a101d0"}
Oct 30 23:46:38.101134 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:38.101078    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" podStartSLOduration=1.270639383 podStartE2EDuration="5.101065239s" totalImagesPullingTime="3.830425856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:38.100908799 +0000 UTC m=+250.447892696" watchObservedRunningTime="2026-10-30 23:46:38.101065239 +0000 UTC m=+250.448049136"
Oct 30 23:46:39.000853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:39.000265    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm" event={"ID":"34e7bbb9-29be-4332-8ddd-34b2e683bd28","Type":"ContainerStarted","Data":"d910dc4191986f0e867fb6ddf93a9e3748e44641b0e0bd5e355a50a9bb7369e3"}
Oct 30 23:46:39.000853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:39.000618    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:39.125391 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:39.125330    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm" podStartSLOduration=1.80633432 podStartE2EDuration="3.125313663s" totalImagesPullingTime="1.318979343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:39.124498873 +0000 UTC m=+251.471482771" watchObservedRunningTime="2026-10-30 23:46:39.125313663 +0000 UTC m=+251.472297560"
Oct 30 23:46:41.010431 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:41.010395    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" event={"ID":"2223dd19-7764-474b-94bb-b8e0d9c94d4c","Type":"ContainerStarted","Data":"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"}
Oct 30 23:46:41.118167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:41.118112    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" podStartSLOduration=2.435901185 podStartE2EDuration="5.118096518s" totalImagesPullingTime="2.682195333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:41.117323929 +0000 UTC m=+253.464307828" watchObservedRunningTime="2026-10-30 23:46:41.118096518 +0000 UTC m=+253.465080413"
Oct 30 23:46:46.967834 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:46:46.967792    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:46:49.001912 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:49.001884    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:50.006310 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:50.006262    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"
Oct 30 23:46:50.162662 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:50.162621    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:50.162915 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:50.162890    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" podUID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" containerName="limitador" containerID="cri-o://d81aba5d0ac694b6bbb707f6711cb8a362e97a0c50d3cf1540061b8bf04fcae0" gracePeriod=30
Oct 30 23:46:51.053447 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.053415    6033 generic.go:356] "Generic (PLEG): container finished" podID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" containerID="d81aba5d0ac694b6bbb707f6711cb8a362e97a0c50d3cf1540061b8bf04fcae0" exitCode=0
Oct 30 23:46:51.053826 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.053494    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" event={"ID":"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0","Type":"ContainerDied","Data":"d81aba5d0ac694b6bbb707f6711cb8a362e97a0c50d3cf1540061b8bf04fcae0"}
Oct 30 23:46:51.105336 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.105309    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:51.216865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.216824    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\" (UniqueName: \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\") pod \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") "
Oct 30 23:46:51.217091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.216914    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\" (UniqueName: \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\") pod \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\" (UID: \"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0\") "
Oct 30 23:46:51.217334 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.217310    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file" pod "aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" (UID: "aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:51.219546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.219513    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w" pod "aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" (UID: "aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0"). InnerVolumeSpecName "kube-api-access-lsf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:51.318247 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.318205    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsf2w\" (UniqueName: \"kubernetes.io/projected/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-kube-api-access-lsf2w\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:51.318247 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:51.318239    6033 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0-config-file\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.059184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:52.059146    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw" event={"ID":"aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0","Type":"ContainerDied","Data":"c2660abe094c416e22aa487c62c0d430f2eb666bf05dacb8cc77c0ac7c09e6af"}
Oct 30 23:46:52.059667 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:52.059218    6033 scope.go:122] "RemoveContainer" containerID="d81aba5d0ac694b6bbb707f6711cb8a362e97a0c50d3cf1540061b8bf04fcae0"
Oct 30 23:46:52.059667 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:52.059154    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"
Oct 30 23:46:52.223689 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:52.223611    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:52.270173 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:52.270139    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r4kdw"]
Oct 30 23:46:53.827692 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:46:53.827657    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" path="/var/lib/kubelet/pods/aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0/volumes"
Oct 30 23:47:16.968968 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:16.968884    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:47:18.453800 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.453763    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:18.454220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.454187    6033 cpu_manager.go:414] "containerMap: removing container" podUID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" containerName="limitador"
Oct 30 23:47:18.454220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.454201    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" containerName="limitador"
Oct 30 23:47:18.454291 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.454275    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafa7493-b13b-4a1f-b0a0-d7c85ddf80a0" containerName="limitador"
Oct 30 23:47:18.460861 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.460837    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.515058 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.514992    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:47:18.515058 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.515010    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 23:47:18.561518 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.561475    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 23:47:18.561518 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.561492    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zz959\""
Oct 30 23:47:18.561518 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.561475    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:47:18.561816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.561561    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 23:47:18.562731 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.562711    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:18.598408 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.598375    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.598585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.598431    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.598585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.598519    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf6c\" (UniqueName: \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.699841 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.699804    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.700093 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.699867    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf6c\" (UniqueName: \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.700093 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.699944    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.702555 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.702523    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.702670 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.702559    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.757762 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.757727    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf6c\" (UniqueName: \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\") pod \"maas-controller-579cf7c6d-twkr6\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") " pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.772602 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.772576    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:18.906230 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.906198    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 30 23:47:18.909243 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.909225    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:18.966685 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:18.966649    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 30 23:47:19.002367 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.002327    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.002537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.002429    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4r6\" (UniqueName: \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.002537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.002509    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.103734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.103643    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.103905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.103733    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.103905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.103778    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4r6\" (UniqueName: \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.106182 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.106156    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.106290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.106193    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.138753 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.138610    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:19.141944 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:19.141914    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a2458c_b5c0_4699_b789_5657a329afba.slice/crio-93b5f07c7137732d34cc2c53bc570d57f4b4fef2a9c69a12c7ecf5f5ab50f247 WatchSource:0}: Error finding container 93b5f07c7137732d34cc2c53bc570d57f4b4fef2a9c69a12c7ecf5f5ab50f247: Status 404 returned error can't find the container with id 93b5f07c7137732d34cc2c53bc570d57f4b4fef2a9c69a12c7ecf5f5ab50f247
Oct 30 23:47:19.161677 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.161651    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4r6\" (UniqueName: \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\") pod \"maas-controller-5c745b59c4-jcxwj\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") " pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.177467 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.177425    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twkr6" event={"ID":"d7a2458c-b5c0-4699-b789-5657a329afba","Type":"ContainerStarted","Data":"93b5f07c7137732d34cc2c53bc570d57f4b4fef2a9c69a12c7ecf5f5ab50f247"}
Oct 30 23:47:19.221952 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.221924    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:19.587418 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:19.587394    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 30 23:47:19.589856 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:19.589828    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9493ebf_b78f_48b3_acac_04cdfcd9b629.slice/crio-678dcd2e78381de7272c44f07feecba23b0dfe247dc4e27b7742d074268d6bc6 WatchSource:0}: Error finding container 678dcd2e78381de7272c44f07feecba23b0dfe247dc4e27b7742d074268d6bc6: Status 404 returned error can't find the container with id 678dcd2e78381de7272c44f07feecba23b0dfe247dc4e27b7742d074268d6bc6
Oct 30 23:47:20.182849 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:20.182814    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerStarted","Data":"678dcd2e78381de7272c44f07feecba23b0dfe247dc4e27b7742d074268d6bc6"}
Oct 30 23:47:23.201360 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:23.201303    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twkr6" event={"ID":"d7a2458c-b5c0-4699-b789-5657a329afba","Type":"ContainerStarted","Data":"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"}
Oct 30 23:47:23.201834 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:23.201494    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:23.202848 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:23.202822    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerStarted","Data":"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"}
Oct 30 23:47:23.202958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:23.202885    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:23.321347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:23.321298    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-twkr6" podStartSLOduration=1.784810448 podStartE2EDuration="5.321283349s" totalImagesPullingTime="3.536472901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:23.320329283 +0000 UTC m=+295.667313202" watchObservedRunningTime="2026-10-30 23:47:23.321283349 +0000 UTC m=+295.668267247"
Oct 30 23:47:26.197149 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.197087    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" podStartSLOduration=4.702208606 podStartE2EDuration="8.197068528s" totalImagesPullingTime="3.494859922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:23.437731281 +0000 UTC m=+295.784715178" watchObservedRunningTime="2026-10-30 23:47:26.197068528 +0000 UTC m=+298.544052428"
Oct 30 23:47:26.198496 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.198463    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"]
Oct 30 23:47:26.202196 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.202172    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.251142 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.251107    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 23:47:26.251337 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.251170    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 23:47:26.272834 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.272795    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.272834 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.272835    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.273070 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.272920    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.273070 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.272974    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcpv\" (UniqueName: \"kubernetes.io/projected/159c39d8-4d1c-4674-af0d-4d134caaad83-kube-api-access-8kcpv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.297875 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.297830    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 23:47:26.298083 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.297981    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-pm4ll\""
Oct 30 23:47:26.299258 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.299240    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"]
Oct 30 23:47:26.374007 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.373965    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.374007 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.374008    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.374316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.374074    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.374316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.374142    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcpv\" (UniqueName: \"kubernetes.io/projected/159c39d8-4d1c-4674-af0d-4d134caaad83-kube-api-access-8kcpv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.374316 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:26.374161    6033 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 23:47:26.374316 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:26.374164    6033 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 23:47:26.374316 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:26.374236    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source podName:159c39d8-4d1c-4674-af0d-4d134caaad83 nodeName:}" failed. No retries permitted until 2026-10-30 23:47:26.874215406 +0000 UTC m=+299.221199296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-fgsdn" (UID: "159c39d8-4d1c-4674-af0d-4d134caaad83") : configmap "usage-tenancy-proxy-source" not found
Oct 30 23:47:26.374506 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:26.374323    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert podName:159c39d8-4d1c-4674-af0d-4d134caaad83 nodeName:}" failed. No retries permitted until 2026-10-30 23:47:26.874304451 +0000 UTC m=+299.221288330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-fgsdn" (UID: "159c39d8-4d1c-4674-af0d-4d134caaad83") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 23:47:26.374795 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.374774    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.432271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.432233    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcpv\" (UniqueName: \"kubernetes.io/projected/159c39d8-4d1c-4674-af0d-4d134caaad83-kube-api-access-8kcpv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.877974 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.877934    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.878192 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.877996    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.878734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.878710    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/159c39d8-4d1c-4674-af0d-4d134caaad83-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:26.880317 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:26.880287    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/159c39d8-4d1c-4674-af0d-4d134caaad83-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fgsdn\" (UID: \"159c39d8-4d1c-4674-af0d-4d134caaad83\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:27.114869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:27.114832    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:27.479476 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:27.479387    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"]
Oct 30 23:47:27.482793 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:27.482763    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159c39d8_4d1c_4674_af0d_4d134caaad83.slice/crio-9192774c89f06b5f9532daab50e1247cc3001dd21a554d1c17178153b041d9d5 WatchSource:0}: Error finding container 9192774c89f06b5f9532daab50e1247cc3001dd21a554d1c17178153b041d9d5: Status 404 returned error can't find the container with id 9192774c89f06b5f9532daab50e1247cc3001dd21a554d1c17178153b041d9d5
Oct 30 23:47:28.237451 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:28.237365    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn" event={"ID":"159c39d8-4d1c-4674-af0d-4d134caaad83","Type":"ContainerStarted","Data":"9192774c89f06b5f9532daab50e1247cc3001dd21a554d1c17178153b041d9d5"}
Oct 30 23:47:34.212052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.211992    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:34.212537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.212453    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:47:34.487090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.486606    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:34.487090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.486836    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-twkr6" podUID="d7a2458c-b5c0-4699-b789-5657a329afba" containerName="manager" containerID="cri-o://6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701" gracePeriod=30
Oct 30 23:47:34.797157 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.797082    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:34.839379 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.839341    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\" (UniqueName: \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\") pod \"d7a2458c-b5c0-4699-b789-5657a329afba\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") "
Oct 30 23:47:34.839555 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.839432    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\") pod \"d7a2458c-b5c0-4699-b789-5657a329afba\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") "
Oct 30 23:47:34.839555 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.839512    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\") pod \"d7a2458c-b5c0-4699-b789-5657a329afba\" (UID: \"d7a2458c-b5c0-4699-b789-5657a329afba\") "
Oct 30 23:47:34.843433 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.843397    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c" pod "d7a2458c-b5c0-4699-b789-5657a329afba" (UID: "d7a2458c-b5c0-4699-b789-5657a329afba"). InnerVolumeSpecName "kube-api-access-6jf6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:34.844641 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.844598    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert" pod "d7a2458c-b5c0-4699-b789-5657a329afba" (UID: "d7a2458c-b5c0-4699-b789-5657a329afba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:34.844871 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.844849    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs" pod "d7a2458c-b5c0-4699-b789-5657a329afba" (UID: "d7a2458c-b5c0-4699-b789-5657a329afba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:34.941080 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.941042    6033 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-webhook-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:34.941080 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.941079    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jf6c\" (UniqueName: \"kubernetes.io/projected/d7a2458c-b5c0-4699-b789-5657a329afba-kube-api-access-6jf6c\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:34.941326 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:34.941095    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7a2458c-b5c0-4699-b789-5657a329afba-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:47:35.274394 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.274356    6033 generic.go:356] "Generic (PLEG): container finished" podID="d7a2458c-b5c0-4699-b789-5657a329afba" containerID="6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701" exitCode=0
Oct 30 23:47:35.274865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.274434    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-twkr6"
Oct 30 23:47:35.274865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.274432    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twkr6" event={"ID":"d7a2458c-b5c0-4699-b789-5657a329afba","Type":"ContainerDied","Data":"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"}
Oct 30 23:47:35.274865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.274499    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-twkr6" event={"ID":"d7a2458c-b5c0-4699-b789-5657a329afba","Type":"ContainerDied","Data":"93b5f07c7137732d34cc2c53bc570d57f4b4fef2a9c69a12c7ecf5f5ab50f247"}
Oct 30 23:47:35.274865 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.274532    6033 scope.go:122] "RemoveContainer" containerID="6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"
Oct 30 23:47:35.287181 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.287019    6033 scope.go:122] "RemoveContainer" containerID="6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"
Oct 30 23:47:35.287437 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:35.287409    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701\": container with ID starting with 6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701 not found: ID does not exist" containerID="6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"
Oct 30 23:47:35.287576 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.287446    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701"} err="failed to get container status \"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701\": rpc error: code = NotFound desc = could not find container \"6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701\": container with ID starting with 6b07416795be25fe2172ef6617a0260731cc2d8de79f4ab256cb14d3ecddd701 not found: ID does not exist"
Oct 30 23:47:35.430478 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.430440    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:35.432840 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.432805    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-twkr6"]
Oct 30 23:47:35.829969 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:35.829930    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a2458c-b5c0-4699-b789-5657a329afba" path="/var/lib/kubelet/pods/d7a2458c-b5c0-4699-b789-5657a329afba/volumes"
Oct 30 23:47:43.322251 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:43.322213    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn" event={"ID":"159c39d8-4d1c-4674-af0d-4d134caaad83","Type":"ContainerStarted","Data":"c896abe279e245c74fd02311c79877ca1cc690d570163401ae7bb0aca7936ca9"}
Oct 30 23:47:43.322687 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:43.322274    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:43.439215 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:43.439165    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn" podStartSLOduration=2.6832964649999997 podStartE2EDuration="17.439149767s" totalImagesPullingTime="14.755853302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:43.437747967 +0000 UTC m=+315.784731866" watchObservedRunningTime="2026-10-30 23:47:43.439149767 +0000 UTC m=+315.786133682"
Oct 30 23:47:46.969740 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:47:46.969694    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:47:50.124053 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.123997    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-szf2t"]
Oct 30 23:47:50.124529 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.124411    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d7a2458c-b5c0-4699-b789-5657a329afba" containerName="manager"
Oct 30 23:47:50.124529 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.124422    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a2458c-b5c0-4699-b789-5657a329afba" containerName="manager"
Oct 30 23:47:50.124529 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.124487    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a2458c-b5c0-4699-b789-5657a329afba" containerName="manager"
Oct 30 23:47:50.127734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.127715    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.178483 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.178451    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 23:47:50.181992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.181962    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affbc01f-9f16-4c16-89f4-99edc24e8680-cert\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.182173 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.182000    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl8l2\" (UniqueName: \"kubernetes.io/projected/affbc01f-9f16-4c16-89f4-99edc24e8680-kube-api-access-wl8l2\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.223904 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.223858    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8b2qp\""
Oct 30 23:47:50.225130 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.225104    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-szf2t"]
Oct 30 23:47:50.240892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.240859    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:47:50.244520 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.244497    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.283128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.283088    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affbc01f-9f16-4c16-89f4-99edc24e8680-cert\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.283128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.283129    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl8l2\" (UniqueName: \"kubernetes.io/projected/affbc01f-9f16-4c16-89f4-99edc24e8680-kube-api-access-wl8l2\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.283335 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.283165    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwcx9\" (UniqueName: \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.283335 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.283254    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.285614 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.285590    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affbc01f-9f16-4c16-89f4-99edc24e8680-cert\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.294172 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.294143    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 23:47:50.294172 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.294143    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-98rcm\""
Oct 30 23:47:50.299833 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.299797    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:47:50.344361 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.344327    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl8l2\" (UniqueName: \"kubernetes.io/projected/affbc01f-9f16-4c16-89f4-99edc24e8680-kube-api-access-wl8l2\") pod \"kserve-controller-manager-776cc66d47-szf2t\" (UID: \"affbc01f-9f16-4c16-89f4-99edc24e8680\") " pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.344864 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.344842    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6cdml"]
Oct 30 23:47:50.349178 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.349155    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.384759 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.384663    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67e907e-94d3-4135-8e91-418bcb790b6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.384759 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.384720    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhs8\" (UniqueName: \"kubernetes.io/projected/d67e907e-94d3-4135-8e91-418bcb790b6e-kube-api-access-fvhs8\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.384945 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.384855    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwcx9\" (UniqueName: \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.384945 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.384895    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.387586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.387557    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.399380 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.399349    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 23:47:50.399583 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.399566    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s4nr4\""
Oct 30 23:47:50.403504 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.403472    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6cdml"]
Oct 30 23:47:50.442774 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.442741    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:50.445742 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.445711    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwcx9\" (UniqueName: \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\") pod \"llmisvc-controller-manager-869d5dcc5d-7dzst\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.458321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.458292    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-v8nkx"]
Oct 30 23:47:50.463107 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.463083    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.485631 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.485598    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67e907e-94d3-4135-8e91-418bcb790b6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.485631 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.485648    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhs8\" (UniqueName: \"kubernetes.io/projected/d67e907e-94d3-4135-8e91-418bcb790b6e-kube-api-access-fvhs8\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.488213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.488191    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67e907e-94d3-4135-8e91-418bcb790b6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.515879 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.515839    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 23:47:50.516052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.515893    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 23:47:50.516106 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.516022    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8cmt6\""
Oct 30 23:47:50.516978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.516843    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-v8nkx"]
Oct 30 23:47:50.546773 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.546739    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhs8\" (UniqueName: \"kubernetes.io/projected/d67e907e-94d3-4135-8e91-418bcb790b6e-kube-api-access-fvhs8\") pod \"model-serving-api-75bd89fdfc-6cdml\" (UID: \"d67e907e-94d3-4135-8e91-418bcb790b6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.557306 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.557277    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:50.586776 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.586736    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bt6z\" (UniqueName: \"kubernetes.io/projected/81f86f98-23df-4c30-ab28-a7b809c2801a-kube-api-access-4bt6z\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.586928 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.586897    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-metrics-certs\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.586990 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.586968    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-cert\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.666646 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.666539    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:50.688587 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.688550    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-cert\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.688721 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.688615    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bt6z\" (UniqueName: \"kubernetes.io/projected/81f86f98-23df-4c30-ab28-a7b809c2801a-kube-api-access-4bt6z\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.688770 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.688723    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-metrics-certs\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.691166 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.691144    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-cert\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.691289 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.691270    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81f86f98-23df-4c30-ab28-a7b809c2801a-metrics-certs\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.746786 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.746751    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bt6z\" (UniqueName: \"kubernetes.io/projected/81f86f98-23df-4c30-ab28-a7b809c2801a-kube-api-access-4bt6z\") pod \"odh-model-controller-769875db56-v8nkx\" (UID: \"81f86f98-23df-4c30-ab28-a7b809c2801a\") " pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.777985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.777946    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:50.812398 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.812246    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-szf2t"]
Oct 30 23:47:50.815526 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:50.815495    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffbc01f_9f16_4c16_89f4_99edc24e8680.slice/crio-86c855dac84eb087fa4293a5c3453b4d9957664e4ca394dc8387bfb061ec1ad2 WatchSource:0}: Error finding container 86c855dac84eb087fa4293a5c3453b4d9957664e4ca394dc8387bfb061ec1ad2: Status 404 returned error can't find the container with id 86c855dac84eb087fa4293a5c3453b4d9957664e4ca394dc8387bfb061ec1ad2
Oct 30 23:47:50.929447 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:50.929291    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:47:50.932313 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:50.932281    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod56d4f898_80a8_44cb_90a6_83ea3a5bd722.slice/crio-98b76ddc166d14f41721ecd3a73490dc364bc610b1c3e8cd9d51097efa6d4d95 WatchSource:0}: Error finding container 98b76ddc166d14f41721ecd3a73490dc364bc610b1c3e8cd9d51097efa6d4d95: Status 404 returned error can't find the container with id 98b76ddc166d14f41721ecd3a73490dc364bc610b1c3e8cd9d51097efa6d4d95
Oct 30 23:47:51.034345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.034318    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-6cdml"]
Oct 30 23:47:51.036888 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:51.036848    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67e907e_94d3_4135_8e91_418bcb790b6e.slice/crio-6e03a7efbd97be08201e183e6f0b5e640d70b332a1657b6e24386d19da84dc74 WatchSource:0}: Error finding container 6e03a7efbd97be08201e183e6f0b5e640d70b332a1657b6e24386d19da84dc74: Status 404 returned error can't find the container with id 6e03a7efbd97be08201e183e6f0b5e640d70b332a1657b6e24386d19da84dc74
Oct 30 23:47:51.153592 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.153560    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-v8nkx"]
Oct 30 23:47:51.157179 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:47:51.157151    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f86f98_23df_4c30_ab28_a7b809c2801a.slice/crio-58c39da30aacbcc1142a1d7bd7617c5419687d7da11e3bb917940827cf1d639a WatchSource:0}: Error finding container 58c39da30aacbcc1142a1d7bd7617c5419687d7da11e3bb917940827cf1d639a: Status 404 returned error can't find the container with id 58c39da30aacbcc1142a1d7bd7617c5419687d7da11e3bb917940827cf1d639a
Oct 30 23:47:51.360438 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.360398    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-v8nkx" event={"ID":"81f86f98-23df-4c30-ab28-a7b809c2801a","Type":"ContainerStarted","Data":"58c39da30aacbcc1142a1d7bd7617c5419687d7da11e3bb917940827cf1d639a"}
Oct 30 23:47:51.361825 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.361792    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" event={"ID":"56d4f898-80a8-44cb-90a6-83ea3a5bd722","Type":"ContainerStarted","Data":"98b76ddc166d14f41721ecd3a73490dc364bc610b1c3e8cd9d51097efa6d4d95"}
Oct 30 23:47:51.363751 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.363720    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t" event={"ID":"affbc01f-9f16-4c16-89f4-99edc24e8680","Type":"ContainerStarted","Data":"86c855dac84eb087fa4293a5c3453b4d9957664e4ca394dc8387bfb061ec1ad2"}
Oct 30 23:47:51.365102 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:51.365072    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml" event={"ID":"d67e907e-94d3-4135-8e91-418bcb790b6e","Type":"ContainerStarted","Data":"6e03a7efbd97be08201e183e6f0b5e640d70b332a1657b6e24386d19da84dc74"}
Oct 30 23:47:54.334097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:54.334057    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fgsdn"
Oct 30 23:47:55.396337 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.396294    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-v8nkx" event={"ID":"81f86f98-23df-4c30-ab28-a7b809c2801a","Type":"ContainerStarted","Data":"04e0b5e0d7261950091ee0500e6cc23878e062e741bff6116c83e022188c86bf"}
Oct 30 23:47:55.396793 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.396424    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:47:55.398570 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.398539    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" event={"ID":"56d4f898-80a8-44cb-90a6-83ea3a5bd722","Type":"ContainerStarted","Data":"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"}
Oct 30 23:47:55.398704 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.398609    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:47:55.400387 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.400355    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t" event={"ID":"affbc01f-9f16-4c16-89f4-99edc24e8680","Type":"ContainerStarted","Data":"2ce634af0a091a25feb2f432f34862a57f3ca75e8c1e6f849b9aad35f484329b"}
Oct 30 23:47:55.400543 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.400481    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:47:55.402005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.401976    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml" event={"ID":"d67e907e-94d3-4135-8e91-418bcb790b6e","Type":"ContainerStarted","Data":"d9f4672473e770649d92b1f6ebedbf6142e2f031074e8ac75db8e57182ec78c8"}
Oct 30 23:47:55.402146 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.402063    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:47:55.513895 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.512790    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-v8nkx" podStartSLOduration=1.347976399 podStartE2EDuration="5.512777386s" totalImagesPullingTime="4.164800987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:55.511864961 +0000 UTC m=+327.858848860" watchObservedRunningTime="2026-10-30 23:47:55.512777386 +0000 UTC m=+327.859761331"
Oct 30 23:47:55.625639 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.625527    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" podStartSLOduration=2.087996799 podStartE2EDuration="5.625510827s" totalImagesPullingTime="3.537514028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:55.624645105 +0000 UTC m=+327.971629003" watchObservedRunningTime="2026-10-30 23:47:55.625510827 +0000 UTC m=+327.972494724"
Oct 30 23:47:55.736491 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.736430    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml" podStartSLOduration=2.317325721 podStartE2EDuration="5.736415359s" totalImagesPullingTime="3.419089638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:55.73512574 +0000 UTC m=+328.082109638" watchObservedRunningTime="2026-10-30 23:47:55.736415359 +0000 UTC m=+328.083399257"
Oct 30 23:47:55.847092 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:47:55.846461    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t" podStartSLOduration=2.20385225 podStartE2EDuration="5.846440153s" totalImagesPullingTime="3.642587903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:55.846109182 +0000 UTC m=+328.193093081" watchObservedRunningTime="2026-10-30 23:47:55.846440153 +0000 UTC m=+328.193424053"
Oct 30 23:48:03.032537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:03.032498    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:48:06.410325 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:06.410294    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-v8nkx"
Oct 30 23:48:06.412464 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:06.412439    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-6cdml"
Oct 30 23:48:08.048864 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.048831    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:08.051527 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.051500    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:08.103722 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.103689    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:08.158136 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.158102    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78rt\" (UniqueName: \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\") pod \"authorino-6f6944bcdd-wzdql\" (UID: \"efbd1b25-5172-4507-a73d-c4a1614445a1\") " pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:08.259422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.259381    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78rt\" (UniqueName: \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\") pod \"authorino-6f6944bcdd-wzdql\" (UID: \"efbd1b25-5172-4507-a73d-c4a1614445a1\") " pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:08.320129 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.320051    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78rt\" (UniqueName: \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\") pod \"authorino-6f6944bcdd-wzdql\" (UID: \"efbd1b25-5172-4507-a73d-c4a1614445a1\") " pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:08.362975 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.362937    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:08.428649 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.428568    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:08.452901 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.452865    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:08.455664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.455645    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:08.507912 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.507880    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:08.561693 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.561651    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pkrg\" (UniqueName: \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\") pod \"authorino-59ccb5df84-w7zv4\" (UID: \"4d03f414-54fa-44f2-9f34-1902ed07d4f6\") " pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:08.663234 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.663125    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pkrg\" (UniqueName: \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\") pod \"authorino-59ccb5df84-w7zv4\" (UID: \"4d03f414-54fa-44f2-9f34-1902ed07d4f6\") " pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:08.721085 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.721049    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pkrg\" (UniqueName: \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\") pod \"authorino-59ccb5df84-w7zv4\" (UID: \"4d03f414-54fa-44f2-9f34-1902ed07d4f6\") " pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:08.729099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.729072    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:08.731976 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:48:08.731942    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefbd1b25_5172_4507_a73d_c4a1614445a1.slice/crio-6f110f820f4cefdf1a1ab666a69adbdee7cf40e634a6111c5951b1838b15a267 WatchSource:0}: Error finding container 6f110f820f4cefdf1a1ab666a69adbdee7cf40e634a6111c5951b1838b15a267: Status 404 returned error can't find the container with id 6f110f820f4cefdf1a1ab666a69adbdee7cf40e634a6111c5951b1838b15a267
Oct 30 23:48:08.768662 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.768626    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:08.892010 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.891977    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:08.920187 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.920107    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:48:08.922958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.922926    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:08.972205 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.972169    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 23:48:08.976724 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:08.976690    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:48:09.067092 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.067060    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjpp\" (UniqueName: \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.067490 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.067125    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.130543 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.130516    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:09.133640 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:48:09.133601    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d03f414_54fa_44f2_9f34_1902ed07d4f6.slice/crio-edaa577b9e9497e8f5923b10c2463eae3a499749710e2f437d2421e11d5564a3 WatchSource:0}: Error finding container edaa577b9e9497e8f5923b10c2463eae3a499749710e2f437d2421e11d5564a3: Status 404 returned error can't find the container with id edaa577b9e9497e8f5923b10c2463eae3a499749710e2f437d2421e11d5564a3
Oct 30 23:48:09.168041 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.168003    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.168237 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.168146    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjpp\" (UniqueName: \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.170529 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.170483    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.226103 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.226071    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjpp\" (UniqueName: \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\") pod \"authorino-7974659cc8-9c2dn\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") " pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.233749 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.233719    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:48:09.481352 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.481244    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" event={"ID":"efbd1b25-5172-4507-a73d-c4a1614445a1","Type":"ContainerStarted","Data":"6f110f820f4cefdf1a1ab666a69adbdee7cf40e634a6111c5951b1838b15a267"}
Oct 30 23:48:09.482543 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.482511    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" event={"ID":"4d03f414-54fa-44f2-9f34-1902ed07d4f6","Type":"ContainerStarted","Data":"edaa577b9e9497e8f5923b10c2463eae3a499749710e2f437d2421e11d5564a3"}
Oct 30 23:48:09.601487 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:09.601418    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:48:09.604126 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:48:09.604080    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod508eced2_bc4f_4da0_8d52_32f24c1aa7eb.slice/crio-1d033df44ea9b613f416b6a1fa266287ab712ca0bb1b5816ecb936c7b9e94ef6 WatchSource:0}: Error finding container 1d033df44ea9b613f416b6a1fa266287ab712ca0bb1b5816ecb936c7b9e94ef6: Status 404 returned error can't find the container with id 1d033df44ea9b613f416b6a1fa266287ab712ca0bb1b5816ecb936c7b9e94ef6
Oct 30 23:48:10.488408 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.488373    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7974659cc8-9c2dn" event={"ID":"508eced2-bc4f-4da0-8d52-32f24c1aa7eb","Type":"ContainerStarted","Data":"1d033df44ea9b613f416b6a1fa266287ab712ca0bb1b5816ecb936c7b9e94ef6"}
Oct 30 23:48:10.490126 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.490091    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" event={"ID":"efbd1b25-5172-4507-a73d-c4a1614445a1","Type":"ContainerStarted","Data":"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"}
Oct 30 23:48:10.490255 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.490201    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" podUID="efbd1b25-5172-4507-a73d-c4a1614445a1" containerName="authorino" containerID="cri-o://9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1" gracePeriod=30
Oct 30 23:48:10.600010 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.599950    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" podStartSLOduration=0.91296092 podStartE2EDuration="2.599931421s" totalImagesPullingTime="1.686970501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:10.597852515 +0000 UTC m=+342.944836414" watchObservedRunningTime="2026-10-30 23:48:10.599931421 +0000 UTC m=+342.946915319"
Oct 30 23:48:10.755276 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.755246    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:10.884478 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.884445    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\" (UniqueName: \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\") pod \"efbd1b25-5172-4507-a73d-c4a1614445a1\" (UID: \"efbd1b25-5172-4507-a73d-c4a1614445a1\") "
Oct 30 23:48:10.886689 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.886653    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt" pod "efbd1b25-5172-4507-a73d-c4a1614445a1" (UID: "efbd1b25-5172-4507-a73d-c4a1614445a1"). InnerVolumeSpecName "kube-api-access-h78rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:10.986111 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:10.985996    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78rt\" (UniqueName: \"kubernetes.io/projected/efbd1b25-5172-4507-a73d-c4a1614445a1-kube-api-access-h78rt\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:11.497277 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.497233    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7974659cc8-9c2dn" event={"ID":"508eced2-bc4f-4da0-8d52-32f24c1aa7eb","Type":"ContainerStarted","Data":"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"}
Oct 30 23:48:11.499236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.499007    6033 generic.go:356] "Generic (PLEG): container finished" podID="efbd1b25-5172-4507-a73d-c4a1614445a1" containerID="9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1" exitCode=2
Oct 30 23:48:11.499236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.499062    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzdql"
Oct 30 23:48:11.499236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.499061    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" event={"ID":"efbd1b25-5172-4507-a73d-c4a1614445a1","Type":"ContainerDied","Data":"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"}
Oct 30 23:48:11.499236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.499105    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzdql" event={"ID":"efbd1b25-5172-4507-a73d-c4a1614445a1","Type":"ContainerDied","Data":"6f110f820f4cefdf1a1ab666a69adbdee7cf40e634a6111c5951b1838b15a267"}
Oct 30 23:48:11.499236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.499123    6033 scope.go:122] "RemoveContainer" containerID="9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"
Oct 30 23:48:11.501424 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.501398    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" event={"ID":"4d03f414-54fa-44f2-9f34-1902ed07d4f6","Type":"ContainerStarted","Data":"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"}
Oct 30 23:48:11.501520 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.501476    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" podUID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" containerName="authorino" containerID="cri-o://b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf" gracePeriod=30
Oct 30 23:48:11.514755 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.514730    6033 scope.go:122] "RemoveContainer" containerID="9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"
Oct 30 23:48:11.515202 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:48:11.515179    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1\": container with ID starting with 9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1 not found: ID does not exist" containerID="9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"
Oct 30 23:48:11.515291 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.515225    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1"} err="failed to get container status \"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1\": rpc error: code = NotFound desc = could not find container \"9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1\": container with ID starting with 9259c4e052361286ddb364543d3046ca3156130ad669ccb638f387aa19d544d1 not found: ID does not exist"
Oct 30 23:48:11.613608 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.613420    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7974659cc8-9c2dn" podStartSLOduration=2.084059545 podStartE2EDuration="3.613402808s" totalImagesPullingTime="1.529343263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:11.612009491 +0000 UTC m=+343.958993389" watchObservedRunningTime="2026-10-30 23:48:11.613402808 +0000 UTC m=+343.960386706"
Oct 30 23:48:11.642219 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.642181    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:48:11.643841 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.642434    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" podUID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" containerName="authorino" containerID="cri-o://06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae" gracePeriod=30
Oct 30 23:48:11.751616 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.751568    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" podStartSLOduration=2.349102863 podStartE2EDuration="3.751550253s" totalImagesPullingTime="1.40244739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:11.749701328 +0000 UTC m=+344.096685226" watchObservedRunningTime="2026-10-30 23:48:11.751550253 +0000 UTC m=+344.098534151"
Oct 30 23:48:11.772460 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.772431    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:11.860784 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.860752    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:11.868655 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.868619    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzdql"]
Oct 30 23:48:11.894918 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.894892    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\" (UniqueName: \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\") pod \"4d03f414-54fa-44f2-9f34-1902ed07d4f6\" (UID: \"4d03f414-54fa-44f2-9f34-1902ed07d4f6\") "
Oct 30 23:48:11.897418 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.897372    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg" pod "4d03f414-54fa-44f2-9f34-1902ed07d4f6" (UID: "4d03f414-54fa-44f2-9f34-1902ed07d4f6"). InnerVolumeSpecName "kube-api-access-2pkrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:11.905313 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.905283    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:48:11.995898 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.995861    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\" (UniqueName: \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\") pod \"2223dd19-7764-474b-94bb-b8e0d9c94d4c\" (UID: \"2223dd19-7764-474b-94bb-b8e0d9c94d4c\") "
Oct 30 23:48:11.996251 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.996215    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pkrg\" (UniqueName: \"kubernetes.io/projected/4d03f414-54fa-44f2-9f34-1902ed07d4f6-kube-api-access-2pkrg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:11.998097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:11.998070    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r" pod "2223dd19-7764-474b-94bb-b8e0d9c94d4c" (UID: "2223dd19-7764-474b-94bb-b8e0d9c94d4c"). InnerVolumeSpecName "kube-api-access-gk89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:12.097198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.097121    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk89r\" (UniqueName: \"kubernetes.io/projected/2223dd19-7764-474b-94bb-b8e0d9c94d4c-kube-api-access-gk89r\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:48:12.508123 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.508084    6033 generic.go:356] "Generic (PLEG): container finished" podID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" containerID="b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf" exitCode=0
Oct 30 23:48:12.508590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.508134    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59ccb5df84-w7zv4"
Oct 30 23:48:12.508590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.508173    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" event={"ID":"4d03f414-54fa-44f2-9f34-1902ed07d4f6","Type":"ContainerDied","Data":"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"}
Oct 30 23:48:12.508590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.508218    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59ccb5df84-w7zv4" event={"ID":"4d03f414-54fa-44f2-9f34-1902ed07d4f6","Type":"ContainerDied","Data":"edaa577b9e9497e8f5923b10c2463eae3a499749710e2f437d2421e11d5564a3"}
Oct 30 23:48:12.508590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.508240    6033 scope.go:122] "RemoveContainer" containerID="b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"
Oct 30 23:48:12.509632 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.509603    6033 generic.go:356] "Generic (PLEG): container finished" podID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" containerID="06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae" exitCode=0
Oct 30 23:48:12.509740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.509655    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks"
Oct 30 23:48:12.509740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.509669    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" event={"ID":"2223dd19-7764-474b-94bb-b8e0d9c94d4c","Type":"ContainerDied","Data":"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"}
Oct 30 23:48:12.509740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.509710    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xfdks" event={"ID":"2223dd19-7764-474b-94bb-b8e0d9c94d4c","Type":"ContainerDied","Data":"d082ff3a51cfb21f79e021328dd766d68e2c8593f50cf77bfb422d2520e4496d"}
Oct 30 23:48:12.518084 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.518065    6033 scope.go:122] "RemoveContainer" containerID="b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"
Oct 30 23:48:12.518429 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:48:12.518409    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf\": container with ID starting with b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf not found: ID does not exist" containerID="b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"
Oct 30 23:48:12.518525 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.518439    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf"} err="failed to get container status \"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf\": rpc error: code = NotFound desc = could not find container \"b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf\": container with ID starting with b08d69bb377ab61e6fbdc721098cd194f1a6aa5662c6c15530739cfff923a6cf not found: ID does not exist"
Oct 30 23:48:12.518525 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.518457    6033 scope.go:122] "RemoveContainer" containerID="06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"
Oct 30 23:48:12.526931 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.526905    6033 scope.go:122] "RemoveContainer" containerID="06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"
Oct 30 23:48:12.527294 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:48:12.527274    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae\": container with ID starting with 06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae not found: ID does not exist" containerID="06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"
Oct 30 23:48:12.527352 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.527304    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae"} err="failed to get container status \"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae\": rpc error: code = NotFound desc = could not find container \"06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae\": container with ID starting with 06234df6c25292323a9f0505f96b5010c859c78524d2ddc125050f3da1cec1ae not found: ID does not exist"
Oct 30 23:48:12.663332 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.663297    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:48:12.666534 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.666482    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xfdks"]
Oct 30 23:48:12.778889 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.778855    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:12.782632 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:12.782603    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59ccb5df84-w7zv4"]
Oct 30 23:48:13.828148 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:13.828104    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" path="/var/lib/kubelet/pods/2223dd19-7764-474b-94bb-b8e0d9c94d4c/volumes"
Oct 30 23:48:13.828651 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:13.828568    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" path="/var/lib/kubelet/pods/4d03f414-54fa-44f2-9f34-1902ed07d4f6/volumes"
Oct 30 23:48:13.828943 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:13.828923    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbd1b25-5172-4507-a73d-c4a1614445a1" path="/var/lib/kubelet/pods/efbd1b25-5172-4507-a73d-c4a1614445a1/volumes"
Oct 30 23:48:16.970098 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:48:16.970052    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:48:26.409846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:26.409813    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:48:26.412842 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:48:26.412817    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-szf2t"
Oct 30 23:48:46.969780 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:48:46.969688    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:49:13.142168 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142131    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-z482g"]
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142546    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142556    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142587    6033 cpu_manager.go:414] "containerMap: removing container" podUID="efbd1b25-5172-4507-a73d-c4a1614445a1" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142592    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbd1b25-5172-4507-a73d-c4a1614445a1" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142602    6033 cpu_manager.go:414] "containerMap: removing container" podUID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142606    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142666    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbd1b25-5172-4507-a73d-c4a1614445a1" containerName="authorino"
Oct 30 23:49:13.142672 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142673    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="2223dd19-7764-474b-94bb-b8e0d9c94d4c" containerName="authorino"
Oct 30 23:49:13.142980 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.142681    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d03f414-54fa-44f2-9f34-1902ed07d4f6" containerName="authorino"
Oct 30 23:49:13.145747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.145727    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.201293 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.201256    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-z482g"]
Oct 30 23:49:13.267711 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.267673    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6znpm\" (UniqueName: \"kubernetes.io/projected/063afbf8-66c3-434a-8e2e-7c9493041154-kube-api-access-6znpm\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.267917 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.267840    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/063afbf8-66c3-434a-8e2e-7c9493041154-cert\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.368657 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.368613    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/063afbf8-66c3-434a-8e2e-7c9493041154-cert\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.368657 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.368661    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6znpm\" (UniqueName: \"kubernetes.io/projected/063afbf8-66c3-434a-8e2e-7c9493041154-kube-api-access-6znpm\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.371200 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.371173    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/063afbf8-66c3-434a-8e2e-7c9493041154-cert\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.426986 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.426900    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6znpm\" (UniqueName: \"kubernetes.io/projected/063afbf8-66c3-434a-8e2e-7c9493041154-kube-api-access-6znpm\") pod \"llmisvc-controller-manager-6466c554d8-z482g\" (UID: \"063afbf8-66c3-434a-8e2e-7c9493041154\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.457952 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.457916    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:13.827504 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:49:13.827468    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063afbf8_66c3_434a_8e2e_7c9493041154.slice/crio-06ec3e2d572acdc5c2d87b6a5cf469087d3815abbd6768518d753595b2d5dc66 WatchSource:0}: Error finding container 06ec3e2d572acdc5c2d87b6a5cf469087d3815abbd6768518d753595b2d5dc66: Status 404 returned error can't find the container with id 06ec3e2d572acdc5c2d87b6a5cf469087d3815abbd6768518d753595b2d5dc66
Oct 30 23:49:13.828209 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:13.828188    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-z482g"]
Oct 30 23:49:14.808104 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:14.808068    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g" event={"ID":"063afbf8-66c3-434a-8e2e-7c9493041154","Type":"ContainerStarted","Data":"4e2d144f05f73656e3a225bf88c7433b49ffa77e66f996e059cde81316902038"}
Oct 30 23:49:14.808530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:14.808123    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g" event={"ID":"063afbf8-66c3-434a-8e2e-7c9493041154","Type":"ContainerStarted","Data":"06ec3e2d572acdc5c2d87b6a5cf469087d3815abbd6768518d753595b2d5dc66"}
Oct 30 23:49:14.808530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:14.808143    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:14.917813 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:14.917754    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g" podStartSLOduration=1.067378312 podStartE2EDuration="1.917738305s" totalImagesPullingTime="850.359993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:14.917106474 +0000 UTC m=+407.264090372" watchObservedRunningTime="2026-10-30 23:49:14.917738305 +0000 UTC m=+407.264722276"
Oct 30 23:49:16.970190 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:49:16.970145    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:49:45.815119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:45.815082    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-z482g"
Oct 30 23:49:45.973349 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:45.973314    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:49:45.973643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:45.973609    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" podUID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" containerName="manager" containerID="cri-o://33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09" gracePeriod=30
Oct 30 23:49:46.219444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.219418    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:49:46.276846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.276817    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\" (UniqueName: \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\") pod \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") "
Oct 30 23:49:46.277078 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.276926    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\" (UniqueName: \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\") pod \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\" (UID: \"56d4f898-80a8-44cb-90a6-83ea3a5bd722\") "
Oct 30 23:49:46.278919 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.278891    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert" pod "56d4f898-80a8-44cb-90a6-83ea3a5bd722" (UID: "56d4f898-80a8-44cb-90a6-83ea3a5bd722"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:49:46.278919 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.278892    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9" pod "56d4f898-80a8-44cb-90a6-83ea3a5bd722" (UID: "56d4f898-80a8-44cb-90a6-83ea3a5bd722"). InnerVolumeSpecName "kube-api-access-xwcx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:46.378623 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.378547    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwcx9\" (UniqueName: \"kubernetes.io/projected/56d4f898-80a8-44cb-90a6-83ea3a5bd722-kube-api-access-xwcx9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:46.378623 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.378574    6033 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56d4f898-80a8-44cb-90a6-83ea3a5bd722-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:49:46.956955 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.956920    6033 generic.go:356] "Generic (PLEG): container finished" podID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" containerID="33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09" exitCode=0
Oct 30 23:49:46.957441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.956971    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"
Oct 30 23:49:46.957441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.957020    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" event={"ID":"56d4f898-80a8-44cb-90a6-83ea3a5bd722","Type":"ContainerDied","Data":"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"}
Oct 30 23:49:46.957441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.957111    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst" event={"ID":"56d4f898-80a8-44cb-90a6-83ea3a5bd722","Type":"ContainerDied","Data":"98b76ddc166d14f41721ecd3a73490dc364bc610b1c3e8cd9d51097efa6d4d95"}
Oct 30 23:49:46.957441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.957157    6033 scope.go:122] "RemoveContainer" containerID="33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"
Oct 30 23:49:46.967546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.967527    6033 scope.go:122] "RemoveContainer" containerID="33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"
Oct 30 23:49:46.967882 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:49:46.967853    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09\": container with ID starting with 33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09 not found: ID does not exist" containerID="33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"
Oct 30 23:49:46.967952 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:46.967895    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09"} err="failed to get container status \"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09\": rpc error: code = NotFound desc = could not find container \"33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09\": container with ID starting with 33bb4f3aff5d45cae1f2a50baf7dbb3fc25d4c42262847d442fdd02b91472a09 not found: ID does not exist"
Oct 30 23:49:46.971612 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:49:46.971581    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:49:47.121229 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:47.121194    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:49:47.167752 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:47.167717    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-7dzst"]
Oct 30 23:49:47.827965 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:47.827931    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" path="/var/lib/kubelet/pods/56d4f898-80a8-44cb-90a6-83ea3a5bd722/volumes"
Oct 30 23:49:51.983412 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:51.983377    6033 generic.go:356] "Generic (PLEG): container finished" podID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerID="195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9" exitCode=1
Oct 30 23:49:51.983845 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:51.983448    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerDied","Data":"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"}
Oct 30 23:49:51.983845 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:51.983796    6033 scope.go:122] "RemoveContainer" containerID="195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"
Oct 30 23:49:53.997405 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:53.997370    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerStarted","Data":"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"}
Oct 30 23:49:53.997814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:49:53.997608    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:50:05.004636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:05.004604    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 30 23:50:09.087464 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:09.087369    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:50:16.969895 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:50:16.969852    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:50:21.840391 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:21.840347    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:50:35.758575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:35.758528    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:50:46.970865 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:50:46.970822    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:50:48.940212 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:48.940176    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:50:57.545420 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.545380    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:50:57.545982 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.545577    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7974659cc8-9c2dn" podUID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" containerName="authorino" containerID="cri-o://c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3" gracePeriod=30
Oct 30 23:50:57.790286 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.790259    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:50:57.892737 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.892649    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\" (UniqueName: \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\") pod \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") "
Oct 30 23:50:57.892913 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.892775    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\" (UniqueName: \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\") pod \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\" (UID: \"508eced2-bc4f-4da0-8d52-32f24c1aa7eb\") "
Oct 30 23:50:57.894872 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.894837    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp" pod "508eced2-bc4f-4da0-8d52-32f24c1aa7eb" (UID: "508eced2-bc4f-4da0-8d52-32f24c1aa7eb"). InnerVolumeSpecName "kube-api-access-xcjpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:50:57.905187 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.905156    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert" pod "508eced2-bc4f-4da0-8d52-32f24c1aa7eb" (UID: "508eced2-bc4f-4da0-8d52-32f24c1aa7eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:50:57.993800 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.993752    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjpp\" (UniqueName: \"kubernetes.io/projected/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-kube-api-access-xcjpp\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:57.993800 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:57.993787    6033 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/508eced2-bc4f-4da0-8d52-32f24c1aa7eb-tls-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:50:58.311455 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.311420    6033 generic.go:356] "Generic (PLEG): container finished" podID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" containerID="c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3" exitCode=0
Oct 30 23:50:58.311641 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.311466    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7974659cc8-9c2dn"
Oct 30 23:50:58.311641 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.311462    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7974659cc8-9c2dn" event={"ID":"508eced2-bc4f-4da0-8d52-32f24c1aa7eb","Type":"ContainerDied","Data":"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"}
Oct 30 23:50:58.311641 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.311517    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7974659cc8-9c2dn" event={"ID":"508eced2-bc4f-4da0-8d52-32f24c1aa7eb","Type":"ContainerDied","Data":"1d033df44ea9b613f416b6a1fa266287ab712ca0bb1b5816ecb936c7b9e94ef6"}
Oct 30 23:50:58.311641 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.311539    6033 scope.go:122] "RemoveContainer" containerID="c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"
Oct 30 23:50:58.320919 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.320896    6033 scope.go:122] "RemoveContainer" containerID="c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"
Oct 30 23:50:58.321283 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:50:58.321256    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3\": container with ID starting with c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3 not found: ID does not exist" containerID="c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"
Oct 30 23:50:58.321339 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.321292    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3"} err="failed to get container status \"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3\": rpc error: code = NotFound desc = could not find container \"c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3\": container with ID starting with c6af195c09e094ad77a3c980d12dcce2e80b53b855f790dc27b65d412fb3cfd3 not found: ID does not exist"
Oct 30 23:50:58.465172 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.465135    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:50:58.468948 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:58.468912    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7974659cc8-9c2dn"]
Oct 30 23:50:59.827759 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.827717    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" path="/var/lib/kubelet/pods/508eced2-bc4f-4da0-8d52-32f24c1aa7eb/volumes"
Oct 30 23:50:59.872703 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.872667    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:50:59.873127 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873111    6033 cpu_manager.go:414] "containerMap: removing container" podUID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" containerName="manager"
Oct 30 23:50:59.873184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873129    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" containerName="manager"
Oct 30 23:50:59.873184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873148    6033 cpu_manager.go:414] "containerMap: removing container" podUID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" containerName="authorino"
Oct 30 23:50:59.873184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873153    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" containerName="authorino"
Oct 30 23:50:59.873290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873224    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d4f898-80a8-44cb-90a6-83ea3a5bd722" containerName="manager"
Oct 30 23:50:59.873290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.873234    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="508eced2-bc4f-4da0-8d52-32f24c1aa7eb" containerName="authorino"
Oct 30 23:50:59.877905 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.877883    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:50:59.973603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.973570    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:50:59.973775 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.973605    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\""
Oct 30 23:50:59.975514 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:50:59.975483    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:51:00.018009 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.017971    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018009 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018014    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018108    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4km\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018166    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018186    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018434 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018330    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018434 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018380    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018523 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018431    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.018523 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.018456    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119503    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119559    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119586    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119893 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119700    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119893 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119787    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119893 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119820    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.119893 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119839    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4km\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119939    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.119987    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.120057    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.120087    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.120119    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120456 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.120436    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.120720 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.120699    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.121957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.121936    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.122108 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.122090    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.177997 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.177961    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.178183 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.178086    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4km\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\") pod \"maas-default-gateway-openshift-default-8f6f87f59-x5f97\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.192930 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.192901    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:00.608412 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.608386    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:51:00.611477 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:51:00.611443    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cb77cf_46c7_47dc_8561_1d3b01804105.slice/crio-98dc72817ed62c7e98acad95e656136c98cf9261b4941e2f4cfb95a08f42bf29 WatchSource:0}: Error finding container 98dc72817ed62c7e98acad95e656136c98cf9261b4941e2f4cfb95a08f42bf29: Status 404 returned error can't find the container with id 98dc72817ed62c7e98acad95e656136c98cf9261b4941e2f4cfb95a08f42bf29
Oct 30 23:51:00.613448 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:00.613432    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:51:01.328062 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:01.327998    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" event={"ID":"46cb77cf-46c7-47dc-8561-1d3b01804105","Type":"ContainerStarted","Data":"98dc72817ed62c7e98acad95e656136c98cf9261b4941e2f4cfb95a08f42bf29"}
Oct 30 23:51:04.751465 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:04.751430    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:04.751733 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:04.751504    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:04.751733 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:04.751535    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:05.348778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:05.348735    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" event={"ID":"46cb77cf-46c7-47dc-8561-1d3b01804105","Type":"ContainerStarted","Data":"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"}
Oct 30 23:51:05.544857 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:05.544797    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podStartSLOduration=2.40792146 podStartE2EDuration="6.544782409s" totalImagesPullingTime="4.136860949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:05.512951737 +0000 UTC m=+517.859935635" watchObservedRunningTime="2026-10-30 23:51:05.544782409 +0000 UTC m=+517.891766307"
Oct 30 23:51:06.202697 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:06.202660    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:06.206176 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:06.206144    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:06.206321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:06.206211    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:07.201054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:07.200494    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:07.201054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:07.200558    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:08.204579 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:08.204537    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:08.205227 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:08.205188    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:09.206536 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:09.206488    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:09.206958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:09.206561    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:10.200066 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:10.199463    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:10.200066 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:10.199530    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:11.193588 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:11.193553    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:11.193975 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:11.193609    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:12.193949 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:12.193908    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-x5f97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:51:12.194475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:12.193983    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 23:51:13.197702 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:13.197671    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:13.198207 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:13.198055    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:13.198730 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:13.198711    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:51:15.012114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.012080    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:51:15.015429 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.015410    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.069892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.069850    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070098 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.069903    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjq8c\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070098 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.069931    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070098 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070081    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070266 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070123    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070266 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070238    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070274    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070311    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.070345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.070332    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.107248 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.107216    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:51:15.171381 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171344    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171381 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171381    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171400    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171417    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171556    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171603    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjq8c\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171634    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171912 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171729    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171912 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171772    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.171912 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.171833    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.172117 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.172096    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.172182 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.172150    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.172337 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.172312    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.172410 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.172375    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.173956 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.173934    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.174361 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.174337    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.229584 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.229550    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.229718 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.229611    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjq8c\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\") pod \"maas-default-gateway-openshift-default-8f6f87f59-mkvqm\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.328792 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.328714    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:15.748721 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.748639    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:51:15.781114 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:51:15.781085    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c63bf8_0ff8_47b7_87d7_ffa7a903be86.slice/crio-fa9699b065dc25029a402a3247e056d5d0edb946126909edca81a96b7b3c74e5 WatchSource:0}: Error finding container fa9699b065dc25029a402a3247e056d5d0edb946126909edca81a96b7b3c74e5: Status 404 returned error can't find the container with id fa9699b065dc25029a402a3247e056d5d0edb946126909edca81a96b7b3c74e5
Oct 30 23:51:15.784176 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.784136    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:15.784260 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.784207    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:15.784260 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:15.784239    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:51:16.406169 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:16.406120    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" event={"ID":"46c63bf8-0ff8-47b7-87d7-ffa7a903be86","Type":"ContainerStarted","Data":"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"}
Oct 30 23:51:16.406705 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:16.406201    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" event={"ID":"46c63bf8-0ff8-47b7-87d7-ffa7a903be86","Type":"ContainerStarted","Data":"fa9699b065dc25029a402a3247e056d5d0edb946126909edca81a96b7b3c74e5"}
Oct 30 23:51:16.569969 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:16.569909    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podStartSLOduration=2.569895805 podStartE2EDuration="2.569895805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:16.568920712 +0000 UTC m=+528.915904644" watchObservedRunningTime="2026-10-30 23:51:16.569895805 +0000 UTC m=+528.916879703"
Oct 30 23:51:16.968582 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:51:16.968544    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:51:17.329533 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:17.329494    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:17.331099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:17.331071    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:17.331255 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:17.331122    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:18.329951 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:18.329918    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:18.330376 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:18.329971    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:19.329167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:19.329130    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:19.329347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:19.329187    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:20.329619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:20.329580    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:20.330142 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:20.329648    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:21.329393 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:21.329354    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:21.329585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:21.329415    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:22.329475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:22.329438    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:22.329960 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:22.329500    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:23.329500 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:23.329464    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-mkvqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:51:23.329874 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:23.329519    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 23:51:24.333237 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:24.333207    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:24.333618 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:24.333574    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:24.334240 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:51:24.334218    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:51:46.968968 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:51:46.968890    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:52:11.548115 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.548077    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 30 23:52:11.551860 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.551836    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.637640 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.637595    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bng\" (UniqueName: \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.637822 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.637660    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.647431 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.647406    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-35a187-dockercfg-jgq68\""
Oct 30 23:52:11.647539 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.647501    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-35a187\""
Oct 30 23:52:11.648817 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.648794    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 30 23:52:11.739231 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.739191    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bng\" (UniqueName: \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.739422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.739248    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.740002 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.739981    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.797220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.797187    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bng\" (UniqueName: \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\") pod \"payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:11.865499 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:11.865406    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:12.248864 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:12.248736    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 30 23:52:12.251402 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:12.251375    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7387d560_e431_4455_82ea_20cae348ed33.slice/crio-d98947e2f670303c6db7288e36e1a0a64d34f2bc596a278137c3368856d2f2ee WatchSource:0}: Error finding container d98947e2f670303c6db7288e36e1a0a64d34f2bc596a278137c3368856d2f2ee: Status 404 returned error can't find the container with id d98947e2f670303c6db7288e36e1a0a64d34f2bc596a278137c3368856d2f2ee
Oct 30 23:52:12.680347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:12.680314    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" event={"ID":"7387d560-e431-4455-82ea-20cae348ed33","Type":"ContainerStarted","Data":"d98947e2f670303c6db7288e36e1a0a64d34f2bc596a278137c3368856d2f2ee"}
Oct 30 23:52:13.482463 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:13.482432    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:52:15.697914 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.697821    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" event={"ID":"7387d560-e431-4455-82ea-20cae348ed33","Type":"ContainerStarted","Data":"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"}
Oct 30 23:52:15.698301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.697915    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:15.854759 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.854697    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" podStartSLOduration=1.8181881469999999 podStartE2EDuration="4.854678844s" totalImagesPullingTime="3.036490697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:15.807209887 +0000 UTC m=+588.154193797" watchObservedRunningTime="2026-10-30 23:52:15.854678844 +0000 UTC m=+588.201662742"
Oct 30 23:52:15.898574 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.898547    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 30 23:52:15.902303 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.902280    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:15.951960 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.951864    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-fe01fa\""
Oct 30 23:52:15.951960 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.951909    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-fe01fa-dockercfg-78qnz\""
Oct 30 23:52:15.958699 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.958137    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 30 23:52:15.982463 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.982420    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7pg\" (UniqueName: \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:15.982639 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:15.982472    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.083583 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.083544    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7pg\" (UniqueName: \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.083767 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.083590    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.084373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.084353    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.141696 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.141660    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7pg\" (UniqueName: \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\") pod \"payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.216004 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.215907    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:16.581775 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.581742    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 30 23:52:16.585166 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:16.585128    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51be1a0_9642_4530_9fe7_4d0239ea5876.slice/crio-3671ed379100d2a82aa7d955da4cd9bb7f28004274eb85da9ecf0acec4f77c24 WatchSource:0}: Error finding container 3671ed379100d2a82aa7d955da4cd9bb7f28004274eb85da9ecf0acec4f77c24: Status 404 returned error can't find the container with id 3671ed379100d2a82aa7d955da4cd9bb7f28004274eb85da9ecf0acec4f77c24
Oct 30 23:52:16.704549 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.704513    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" event={"ID":"c51be1a0-9642-4530-9fe7-4d0239ea5876","Type":"ContainerStarted","Data":"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"}
Oct 30 23:52:16.705060 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.704574    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" event={"ID":"c51be1a0-9642-4530-9fe7-4d0239ea5876","Type":"ContainerStarted","Data":"3671ed379100d2a82aa7d955da4cd9bb7f28004274eb85da9ecf0acec4f77c24"}
Oct 30 23:52:16.814275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:16.814224    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" podStartSLOduration=1.814206741 podStartE2EDuration="1.814206741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:16.812539919 +0000 UTC m=+589.159523817" watchObservedRunningTime="2026-10-30 23:52:16.814206741 +0000 UTC m=+589.161190638"
Oct 30 23:52:16.972465 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:52:16.972386    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:52:17.708946 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:17.708915    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:18.337172 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.337137    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 30 23:52:18.340892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.340869    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.390162 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.390127    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-34f1ce\""
Oct 30 23:52:18.436942 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.436907    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-34f1ce-dockercfg-nvghw\""
Oct 30 23:52:18.438423 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.438396    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 30 23:52:18.507243 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.507208    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.507424 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.507255    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmvdm\" (UniqueName: \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.509806 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.509776    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 30 23:52:18.513411 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.513390    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.568001 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.567968    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 30 23:52:18.608307 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.608206    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.608307 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.608256    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmvdm\" (UniqueName: \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.608307 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.608287    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.608600 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.608317    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gks\" (UniqueName: \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.609102 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.609078    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.666379 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.666340    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmvdm\" (UniqueName: \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\") pod \"payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:18.709419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.709387    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.709904 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.709444    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gks\" (UniqueName: \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.710246 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.710219    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.767669 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.767634    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gks\" (UniqueName: \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\") pod \"payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.826435 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.826400    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:18.954059 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:18.953991    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:19.191869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.191834    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 30 23:52:19.194710 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:19.194677    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0181a2c_cd33_450a_a2cc_c59030161f7b.slice/crio-663886dfff63d78bac2f1f45dce72461e8e14b14d1de8ce47a60cadba31cd2ce WatchSource:0}: Error finding container 663886dfff63d78bac2f1f45dce72461e8e14b14d1de8ce47a60cadba31cd2ce: Status 404 returned error can't find the container with id 663886dfff63d78bac2f1f45dce72461e8e14b14d1de8ce47a60cadba31cd2ce
Oct 30 23:52:19.325152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.325127    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 30 23:52:19.327686 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:19.327646    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d815df5_b09e_41ff_9eb9_8fc6684dc238.slice/crio-fbfaad911bbf8d72e4b762aaddaa3096dcfdb603f38b26790fa918f88ad4b886 WatchSource:0}: Error finding container fbfaad911bbf8d72e4b762aaddaa3096dcfdb603f38b26790fa918f88ad4b886: Status 404 returned error can't find the container with id fbfaad911bbf8d72e4b762aaddaa3096dcfdb603f38b26790fa918f88ad4b886
Oct 30 23:52:19.719455 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.719414    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" event={"ID":"c0181a2c-cd33-450a-a2cc-c59030161f7b","Type":"ContainerStarted","Data":"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"}
Oct 30 23:52:19.719900 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.719492    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:19.719900 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.719516    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" event={"ID":"c0181a2c-cd33-450a-a2cc-c59030161f7b","Type":"ContainerStarted","Data":"663886dfff63d78bac2f1f45dce72461e8e14b14d1de8ce47a60cadba31cd2ce"}
Oct 30 23:52:19.721101 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.721061    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" event={"ID":"8d815df5-b09e-41ff-9eb9-8fc6684dc238","Type":"ContainerStarted","Data":"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"}
Oct 30 23:52:19.721242 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.721124    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" event={"ID":"8d815df5-b09e-41ff-9eb9-8fc6684dc238","Type":"ContainerStarted","Data":"fbfaad911bbf8d72e4b762aaddaa3096dcfdb603f38b26790fa918f88ad4b886"}
Oct 30 23:52:19.721242 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.721149    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:19.830963 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.830913    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" podStartSLOduration=1.830898052 podStartE2EDuration="1.830898052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:19.829674428 +0000 UTC m=+592.176658338" watchObservedRunningTime="2026-10-30 23:52:19.830898052 +0000 UTC m=+592.177881951"
Oct 30 23:52:19.940277 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:19.940219    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" podStartSLOduration=1.94020525 podStartE2EDuration="1.94020525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:19.938539024 +0000 UTC m=+592.285522960" watchObservedRunningTime="2026-10-30 23:52:19.94020525 +0000 UTC m=+592.287189147"
Oct 30 23:52:22.977924 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:22.977886    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:52:23.038094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.038066    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 30 23:52:23.042149 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.042119    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.091221 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.091190    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-46b586\""
Oct 30 23:52:23.137716 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.137687    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-46b586-dockercfg-jp427\""
Oct 30 23:52:23.139105 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.139083    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 30 23:52:23.161760 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.161718    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgzb\" (UniqueName: \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.161760 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.161763    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.263315 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.263280    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgzb\" (UniqueName: \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.263315 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.263318    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.264099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.264078    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.322397 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.322366    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgzb\" (UniqueName: \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\") pod \"payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.356365 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.356331    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:23.725285 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.725250    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 30 23:52:23.727616 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:23.727575    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e62af63_1817_44c7_b273_2ec390b0eb94.slice/crio-778162e5fd8ba84f050fdc64466bd75181ead84eb5798f418a6dcb77cc2fd36f WatchSource:0}: Error finding container 778162e5fd8ba84f050fdc64466bd75181ead84eb5798f418a6dcb77cc2fd36f: Status 404 returned error can't find the container with id 778162e5fd8ba84f050fdc64466bd75181ead84eb5798f418a6dcb77cc2fd36f
Oct 30 23:52:23.743960 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:23.743898    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" event={"ID":"0e62af63-1817-44c7-b273-2ec390b0eb94","Type":"ContainerStarted","Data":"778162e5fd8ba84f050fdc64466bd75181ead84eb5798f418a6dcb77cc2fd36f"}
Oct 30 23:52:24.751279 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:24.751245    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" event={"ID":"0e62af63-1817-44c7-b273-2ec390b0eb94","Type":"ContainerStarted","Data":"8612d8b3409ab59b442e0242ca79a7317c6dc0fa3c62559a32508cdf38958e90"}
Oct 30 23:52:24.751670 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:24.751315    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:24.864974 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:24.864916    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" podStartSLOduration=2.864899111 podStartE2EDuration="2.864899111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:24.863901915 +0000 UTC m=+597.210885809" watchObservedRunningTime="2026-10-30 23:52:24.864899111 +0000 UTC m=+597.211883008"
Oct 30 23:52:26.707804 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:26.707777    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 30 23:52:28.716042 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:28.716008    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 30 23:52:30.730069 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:30.730018    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 30 23:52:30.730558 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:30.730455    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 30 23:52:32.092128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:32.092094    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:52:34.041619 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.041584    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 30 23:52:34.046856 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.046826    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.069858 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.069820    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.070039 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.069943    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.070039 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.069975    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.070128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.070094    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcxn\" (UniqueName: \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.143356 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.143321    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2nk58\""
Oct 30 23:52:34.143356 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.143354    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:52:34.143356 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.143331    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-986d37-serving-cert\""
Oct 30 23:52:34.143605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.143374    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:52:34.143605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.143414    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 23:52:34.145073 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.145049    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 30 23:52:34.171150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.171112    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.171376 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.171170    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.171376 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.171258    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcxn\" (UniqueName: \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.171376 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:52:34.171306    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-serving-cert: secret "maas-api-e2e-worker-w3-986d37-serving-cert" not found
Oct 30 23:52:34.171376 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.171328    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.171653 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:52:34.171403    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls podName:f7566f75-16ce-4d60-9361-ccb4f2ef3c28 nodeName:}" failed. No retries permitted until 2026-10-30 23:52:34.671379603 +0000 UTC m=+607.018363498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls") pod "maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" (UID: "f7566f75-16ce-4d60-9361-ccb4f2ef3c28") : secret "maas-api-e2e-worker-w3-986d37-serving-cert" not found
Oct 30 23:52:34.172467 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.172437    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.174306 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.174284    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.229222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.229190    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcxn\" (UniqueName: \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.677559 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.677521    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.680010 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.679977    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") pod \"maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:34.961411 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:34.961312    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:35.330233 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:35.330210    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 30 23:52:35.332283 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:35.332257    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7566f75_16ce_4d60_9361_ccb4f2ef3c28.slice/crio-ac01a232104b306b6dc0e7962372f12aabeb7a50cc182a065305fb481ad57ac4 WatchSource:0}: Error finding container ac01a232104b306b6dc0e7962372f12aabeb7a50cc182a065305fb481ad57ac4: Status 404 returned error can't find the container with id ac01a232104b306b6dc0e7962372f12aabeb7a50cc182a065305fb481ad57ac4
Oct 30 23:52:35.760546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:35.760518    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 30 23:52:35.806916 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:35.806875    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" event={"ID":"f7566f75-16ce-4d60-9361-ccb4f2ef3c28","Type":"ContainerStarted","Data":"ac01a232104b306b6dc0e7962372f12aabeb7a50cc182a065305fb481ad57ac4"}
Oct 30 23:52:37.819869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:37.819779    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" event={"ID":"f7566f75-16ce-4d60-9361-ccb4f2ef3c28","Type":"ContainerStarted","Data":"0eaec6ec55ba8cece4fb7b2eb0b175c03e29b0c28c06f0577a42f5f69ba455f6"}
Oct 30 23:52:37.820249 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:37.819990    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:37.936505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:37.936442    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" podStartSLOduration=1.742669511 podStartE2EDuration="3.936422281s" totalImagesPullingTime="2.19375277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:37.935148824 +0000 UTC m=+610.282132724" watchObservedRunningTime="2026-10-30 23:52:37.936422281 +0000 UTC m=+610.283406180"
Oct 30 23:52:40.742296 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.742255    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 30 23:52:40.746289 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.746269    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.795915 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.795883    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-35a187-openshift-default-dockercfg-4wqw9\""
Oct 30 23:52:40.843396 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843351    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843396 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843387    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843649 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843506    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843649 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843628    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843754 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843656    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843754 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843748    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843849 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843787    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlp2\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843849 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843810    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.843928 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.843850    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.845124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.845092    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 30 23:52:40.945072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945009    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945082    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945133    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945162    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlp2\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945186    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945214    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945238    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945271 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945253    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945635    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945642    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945700 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945681    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945710    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945759    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.945970 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.945927    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.947648 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.947623    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:40.947892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:40.947873    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:41.007975 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.007945    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:41.008144 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.008098    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlp2\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:41.060144 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.060107    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:41.479603 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.479578    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 30 23:52:41.482200 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:41.482163    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a2b4d1_a073_445a_910c_fb23a8de8faa.slice/crio-52cf5d61eeb32ff21dae83de17c0e0643be020a48fa02f926c2c0424e3f75481 WatchSource:0}: Error finding container 52cf5d61eeb32ff21dae83de17c0e0643be020a48fa02f926c2c0424e3f75481: Status 404 returned error can't find the container with id 52cf5d61eeb32ff21dae83de17c0e0643be020a48fa02f926c2c0424e3f75481
Oct 30 23:52:41.484747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.484715    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:41.484816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.484786    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:41.484850 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.484814    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:41.842299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.842264    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" event={"ID":"04a2b4d1-a073-445a-910c-fb23a8de8faa","Type":"ContainerStarted","Data":"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"}
Oct 30 23:52:41.842772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.842321    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" event={"ID":"04a2b4d1-a073-445a-910c-fb23a8de8faa","Type":"ContainerStarted","Data":"52cf5d61eeb32ff21dae83de17c0e0643be020a48fa02f926c2c0424e3f75481"}
Oct 30 23:52:41.966311 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:41.966256    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podStartSLOduration=1.966239719 podStartE2EDuration="1.966239719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:41.964480026 +0000 UTC m=+614.311463924" watchObservedRunningTime="2026-10-30 23:52:41.966239719 +0000 UTC m=+614.313223616"
Oct 30 23:52:42.061347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:42.061305    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:42.062805 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:42.062781    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:42.062934 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:42.062836    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:43.060575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:43.060538    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:43.061060 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:43.060593    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:43.830745 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:43.830716    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 30 23:52:44.060906 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:44.060870    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:44.061317 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:44.060919    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:45.060901 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:45.060863    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:45.061107 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:45.060921    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:46.061269 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:46.061233    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:46.061698 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:46.061305    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:46.630490 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:46.630457    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:52:46.969792 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:52:46.969703    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:52:47.061412 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:47.061378    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 30 23:52:47.061800 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:47.061447    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 30 23:52:48.067040 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:48.067000    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:48.067448 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:48.067384    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:48.068090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:48.068072    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 30 23:52:54.238740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.238701    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 30 23:52:54.246586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.246559    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.297715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.297681    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 30 23:52:54.342507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.342475    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"kube-root-ca.crt\""
Oct 30 23:52:54.342507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.342483    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"default-dockercfg-bz7t5\""
Oct 30 23:52:54.342507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.342475    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"openshift-service-ca.crt\""
Oct 30 23:52:54.342507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.342486    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"simulator-w2-27e82b-kserve-self-signed-certs\""
Oct 30 23:52:54.408498 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408465    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.408717 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408623    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.408717 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408691    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.408817 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408734    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.408817 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408753    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj4zz\" (UniqueName: \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.408817 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.408775    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509773 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509740    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509835    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509860    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509896    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509924    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj4zz\" (UniqueName: \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.509957 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.509952    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.510270 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.510233    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.510335 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.510298    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.510414 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.510393    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.512325 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.512299    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.512666 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.512643    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.568612 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.568574    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj4zz\" (UniqueName: \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\") pod \"simulator-w2-27e82b-kserve-776b66d97f-p77pd\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") " pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:54.864136 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:54.864009    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:55.232149 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:55.232120    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 30 23:52:55.234614 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:55.234582    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54ed9707_5cc1_476c_b4f6_1659f402d75b.slice/crio-0c924a352b8d295c5d55b5aef1c1baa521ccbd533b55ad50dac432e511f2923b WatchSource:0}: Error finding container 0c924a352b8d295c5d55b5aef1c1baa521ccbd533b55ad50dac432e511f2923b: Status 404 returned error can't find the container with id 0c924a352b8d295c5d55b5aef1c1baa521ccbd533b55ad50dac432e511f2923b
Oct 30 23:52:55.920228 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:55.920088    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" event={"ID":"54ed9707-5cc1-476c-b4f6-1659f402d75b","Type":"ContainerStarted","Data":"0c924a352b8d295c5d55b5aef1c1baa521ccbd533b55ad50dac432e511f2923b"}
Oct 30 23:52:56.299146 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.299051    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 30 23:52:56.318293 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.317915    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.420061 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.420012    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 30 23:52:56.426777 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.426752    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-46b586-openshift-default-dockercfg-5bltr\""
Oct 30 23:52:56.435324 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435286    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435480 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435429    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435533 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435473    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435626 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435608    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqmw\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435674 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435655    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435720 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435704    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435798 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435777    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435828    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.435961 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.435933    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537022 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.536982    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537095    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537127    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537175    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqmw\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537210    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537275 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537267    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537608 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537290    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537608 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537333    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537608 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537389    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537772 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537605    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537829 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537802    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.537882 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537822    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.538004 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537918    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.538004 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.537924    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.539740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.539690    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.540385 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.540364    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.608333 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.608227    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.608491 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.608379    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqmw\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.635122 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.635085    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:56.930182 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:56.930149    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" event={"ID":"54ed9707-5cc1-476c-b4f6-1659f402d75b","Type":"ContainerStarted","Data":"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"}
Oct 30 23:52:57.044427 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.044377    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podStartSLOduration=1.481927915 podStartE2EDuration="3.044359446s" totalImagesPullingTime="1.562431531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:57.042457773 +0000 UTC m=+629.389441683" watchObservedRunningTime="2026-10-30 23:52:57.044359446 +0000 UTC m=+629.391343343"
Oct 30 23:52:57.083365 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.083331    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 30 23:52:57.093659 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:52:57.093629    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05295f90_3b8c_4406_b5ee_dab044069376.slice/crio-3196c1fd4071f786ed4bfb97d4c08322bbc8acbd38334e99514317848b8a057d WatchSource:0}: Error finding container 3196c1fd4071f786ed4bfb97d4c08322bbc8acbd38334e99514317848b8a057d: Status 404 returned error can't find the container with id 3196c1fd4071f786ed4bfb97d4c08322bbc8acbd38334e99514317848b8a057d
Oct 30 23:52:57.096213 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.096178    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:57.096304 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.096246    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:57.096304 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.096278    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:52:57.864335 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.864298    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:52:57.940360 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.940323    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" event={"ID":"05295f90-3b8c-4406-b5ee-dab044069376","Type":"ContainerStarted","Data":"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"}
Oct 30 23:52:57.940778 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:57.940382    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" event={"ID":"05295f90-3b8c-4406-b5ee-dab044069376","Type":"ContainerStarted","Data":"3196c1fd4071f786ed4bfb97d4c08322bbc8acbd38334e99514317848b8a057d"}
Oct 30 23:52:58.636232 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:58.636199    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:52:58.637717 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:58.637691    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 23:52:58.637829 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:58.637741    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 23:52:59.635984 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:59.635950    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 23:52:59.636442 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:52:59.636003    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 23:53:00.636256 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:00.636219    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 23:53:00.636761 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:00.636299    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 23:53:00.806082 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:00.786792    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podStartSLOduration=4.7867699649999995 podStartE2EDuration="4.786769965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:58.071655039 +0000 UTC m=+630.418638938" watchObservedRunningTime="2026-10-30 23:53:00.786769965 +0000 UTC m=+633.133753863"
Oct 30 23:53:00.807123 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:00.807088    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:53:01.635753 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:01.635709    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 23:53:01.635936 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:01.635789    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 23:53:02.635550 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:02.635508    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 23:53:02.635966 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:02.635565    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 23:53:03.640492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:03.640463    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:53:03.640971 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:03.640865    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:53:03.641530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:03.641509    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 30 23:53:04.864489 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:04.864442    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:53:04.877321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:04.877290    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:53:04.989636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:04.989603    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 30 23:53:11.021420 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.021349    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 30 23:53:11.025826 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.025801    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.110981 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.110938    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.110995    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111126    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111265 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111174    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111256    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111355 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111343    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111390 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111384    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111423 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111410    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.111472 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.111460    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zs7p\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.127852 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.127816    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 30 23:53:11.212222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212187    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212247    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212286    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212311    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212344    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zs7p\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212386    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212688 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212437    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212688 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212474    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212688 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212523    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212688 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212655    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212881 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212789    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.212881 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212865    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.213002 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.212980    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.213124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.213107    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.214667 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.214646    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.214976 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.214956    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.271462 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.271392    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.272754 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.272731    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zs7p\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.340013 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.339977    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:11.761205 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.761048    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 30 23:53:11.763973 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:53:11.763943    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4cecc1_50da_4e73_84e6_656bf87266e1.slice/crio-c84dbca76c2ae229cc5f14cea8bc62d0b986ed0dfee1275dfc6f66dd24ea80f9 WatchSource:0}: Error finding container c84dbca76c2ae229cc5f14cea8bc62d0b986ed0dfee1275dfc6f66dd24ea80f9: Status 404 returned error can't find the container with id c84dbca76c2ae229cc5f14cea8bc62d0b986ed0dfee1275dfc6f66dd24ea80f9
Oct 30 23:53:11.766510 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.766475    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:53:11.766586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.766546    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:53:11.766622 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:11.766579    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:53:12.016369 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.016281    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" event={"ID":"3f4cecc1-50da-4e73-84e6-656bf87266e1","Type":"ContainerStarted","Data":"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"}
Oct 30 23:53:12.016369 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.016338    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" event={"ID":"3f4cecc1-50da-4e73-84e6-656bf87266e1","Type":"ContainerStarted","Data":"c84dbca76c2ae229cc5f14cea8bc62d0b986ed0dfee1275dfc6f66dd24ea80f9"}
Oct 30 23:53:12.135823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.135751    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podStartSLOduration=2.135730038 podStartE2EDuration="2.135730038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:12.133392907 +0000 UTC m=+644.480376805" watchObservedRunningTime="2026-10-30 23:53:12.135730038 +0000 UTC m=+644.482713937"
Oct 30 23:53:12.340461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.340375    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:12.342126 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.342088    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:12.342253 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:12.342142    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:13.341312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:13.341276    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:13.341697 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:13.341346    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:14.341298 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:14.341265    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:14.341554 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:14.341315    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:15.340844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:15.340805    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:15.341056 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:15.340860    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:16.340535 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:16.340501    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:16.340921 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:16.340556    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:16.972826 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:53:16.972766    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:53:17.340379 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:17.340344    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 23:53:17.340566 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:17.340400    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 23:53:18.344991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:18.344960    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:18.345461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:18.345242    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:18.345992 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:18.345975    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 30 23:53:24.555741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:24.555698    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:53:26.730164 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.730106    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 30 23:53:26.735084 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.735057    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.792512 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.792450    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 30 23:53:26.833947 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.833915    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"default-dockercfg-dhj9d\""
Oct 30 23:53:26.834126 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.834063    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"simulator-w1-5c3374-kserve-self-signed-certs\""
Oct 30 23:53:26.856326 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.856287    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"openshift-service-ca.crt\""
Oct 30 23:53:26.856326 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.856294    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"kube-root-ca.crt\""
Oct 30 23:53:26.898629 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898594    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.898858 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898634    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5tbd\" (UniqueName: \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.898858 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898730    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.898858 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898755    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.898858 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898783    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.899134 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.898901    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:26.999983 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:26.999954    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5tbd\" (UniqueName: \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000165 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000017    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000165 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000077    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000165 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000116    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000344 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000186    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000344 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000321    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000565 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000542    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000646 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000605    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.000646 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.000628    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.002453 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.002432    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.002736 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.002715    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.074431 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.074397    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5tbd\" (UniqueName: \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\") pod \"simulator-w1-5c3374-kserve-df759567-nbp9w\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") " pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.348715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.348632    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:27.730888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:27.730863    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 30 23:53:27.733430 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:53:27.733401    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bea949e_21bc_4993_8004_9922b5dbd11b.slice/crio-f8f28ab4949a59e79de06bddf5efd9740ffbfa5672db5c5b480e29229fc34363 WatchSource:0}: Error finding container f8f28ab4949a59e79de06bddf5efd9740ffbfa5672db5c5b480e29229fc34363: Status 404 returned error can't find the container with id f8f28ab4949a59e79de06bddf5efd9740ffbfa5672db5c5b480e29229fc34363
Oct 30 23:53:28.104937 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:28.104901    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" event={"ID":"6bea949e-21bc-4993-8004-9922b5dbd11b","Type":"ContainerStarted","Data":"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"}
Oct 30 23:53:28.104937 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:28.104949    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" event={"ID":"6bea949e-21bc-4993-8004-9922b5dbd11b","Type":"ContainerStarted","Data":"f8f28ab4949a59e79de06bddf5efd9740ffbfa5672db5c5b480e29229fc34363"}
Oct 30 23:53:28.349727 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:28.349686    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:37.349162 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:37.349128    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:37.361930 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:37.361906    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:37.476812 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:37.476752    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podStartSLOduration=11.476735046 podStartE2EDuration="11.476735046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:28.239471291 +0000 UTC m=+660.586455189" watchObservedRunningTime="2026-10-30 23:53:37.476735046 +0000 UTC m=+669.823718943"
Oct 30 23:53:38.169081 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:38.169025    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 30 23:53:42.956464 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:42.956427    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:53:45.999338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:45.999299    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 30 23:53:46.003269 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.003246    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.054291 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.054258    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 30 23:53:46.099183 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.099141    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"simulator-w5-fe01fa-kserve-self-signed-certs\""
Oct 30 23:53:46.100577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.100553    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"default-dockercfg-kv2s5\""
Oct 30 23:53:46.100577 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.100555    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"kube-root-ca.crt\""
Oct 30 23:53:46.100787 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.100744    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fe01fa\"/\"openshift-service-ca.crt\""
Oct 30 23:53:46.102253 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102235    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.102305 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102266    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.102305 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102297    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.102466 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102449    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7pf\" (UniqueName: \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.102595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102575    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.102663 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.102619    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203355 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203314    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203355 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203361    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203616 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203479    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203616 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203520    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203616 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203544    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203786 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203620    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7pf\" (UniqueName: \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.203913 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203881    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.204055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203912    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.204055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.203961    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.205948 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.205918    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.206186 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.206167    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.262176 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.262142    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7pf\" (UniqueName: \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\") pod \"simulator-w5-fe01fa-kserve-699959f59c-fxg9z\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") " pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.315247 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.315203    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:46.698192 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:46.698164    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 30 23:53:46.699648 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:53:46.699617    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2627161_2e28_4343_a4c6_412d8b1c3a00.slice/crio-ce1cb01e831199dfac2478860cff7a69ddbb5a18b7b902b73866e6d12eb02bc5 WatchSource:0}: Error finding container ce1cb01e831199dfac2478860cff7a69ddbb5a18b7b902b73866e6d12eb02bc5: Status 404 returned error can't find the container with id ce1cb01e831199dfac2478860cff7a69ddbb5a18b7b902b73866e6d12eb02bc5
Oct 30 23:53:46.970132 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:53:46.970022    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:53:47.207350 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:47.207309    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" event={"ID":"c2627161-2e28-4343-a4c6-412d8b1c3a00","Type":"ContainerStarted","Data":"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"}
Oct 30 23:53:47.207734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:47.207364    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" event={"ID":"c2627161-2e28-4343-a4c6-412d8b1c3a00","Type":"ContainerStarted","Data":"ce1cb01e831199dfac2478860cff7a69ddbb5a18b7b902b73866e6d12eb02bc5"}
Oct 30 23:53:47.315382 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:47.315355    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:47.320277 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:47.320232    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podStartSLOduration=2.320218805 podStartE2EDuration="2.320218805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:47.319080174 +0000 UTC m=+679.666064070" watchObservedRunningTime="2026-10-30 23:53:47.320218805 +0000 UTC m=+679.667202702"
Oct 30 23:53:56.316128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:56.316077    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:56.329449 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:56.329418    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:53:57.279441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:53:57.279362    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 30 23:54:02.994066 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:02.994007    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:54:13.852903 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.852865    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 30 23:54:13.856738 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.856718    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.906585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.906557    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"premium-w2-27e82b-kserve-self-signed-certs\""
Oct 30 23:54:13.910620 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.910583    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.910926 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.910896    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvngf\" (UniqueName: \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.911101 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.910960    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.911101 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.911061    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.911238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.911125    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.911238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.911159    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:13.913220 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:13.913182    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 30 23:54:14.012448 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012407    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.012630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012471    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.012630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012497    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.012630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012531    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.012630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012576    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvngf\" (UniqueName: \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.012630 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.012615    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.013045 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.013000    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.013181 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.013067    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.013181 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.013089    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.014895 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.014866    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.015042 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.015015    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.071184 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.071150    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvngf\" (UniqueName: \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\") pod \"premium-w2-27e82b-kserve-5654b77f46-977rn\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") " pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.170091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.169974    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:14.551755 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:14.551726    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 30 23:54:14.554276 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:54:14.554246    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5d19655_4df5_42bb_aca8_d782c33fa3df.slice/crio-7851cc97b66b898c1ef3b0ca8273843c712a29dc9312d05027c947305669ae0f WatchSource:0}: Error finding container 7851cc97b66b898c1ef3b0ca8273843c712a29dc9312d05027c947305669ae0f: Status 404 returned error can't find the container with id 7851cc97b66b898c1ef3b0ca8273843c712a29dc9312d05027c947305669ae0f
Oct 30 23:54:15.369152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:15.369119    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" event={"ID":"c5d19655-4df5-42bb-aca8-d782c33fa3df","Type":"ContainerStarted","Data":"1e2921034f5f34810e64757ac698b3fd748019a3999116cb75a508ad8093d500"}
Oct 30 23:54:15.369152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:15.369159    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" event={"ID":"c5d19655-4df5-42bb-aca8-d782c33fa3df","Type":"ContainerStarted","Data":"7851cc97b66b898c1ef3b0ca8273843c712a29dc9312d05027c947305669ae0f"}
Oct 30 23:54:15.486302 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:15.486244    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podStartSLOduration=2.486225115 podStartE2EDuration="2.486225115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:15.485278777 +0000 UTC m=+707.832262675" watchObservedRunningTime="2026-10-30 23:54:15.486225115 +0000 UTC m=+707.833209013"
Oct 30 23:54:16.171097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:16.171063    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:16.968719 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:54:16.968686    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:54:24.170154 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:24.170115    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:24.182814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:24.182774    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:24.432884 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:24.432802    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 30 23:54:24.751321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:24.751294    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:54:46.969676 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:54:46.969586    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:54:47.006212 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:54:47.006172    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:55:06.246723 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:06.246682    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:55:09.061951 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.061913    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 30 23:55:09.066075 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.066052    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.163504 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.163464    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"default-dockercfg-bz6sc\""
Oct 30 23:55:09.163726 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.163705    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"openshift-service-ca.crt\""
Oct 30 23:55:09.163798 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.163715    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"premium-w6-35a187-kserve-self-signed-certs\""
Oct 30 23:55:09.164012 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.163994    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-35a187\"/\"kube-root-ca.crt\""
Oct 30 23:55:09.166585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.166563    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 30 23:55:09.215326 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215249    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.216052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215340    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.216052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215419    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.216052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215478    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsq9\" (UniqueName: \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.216052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215515    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.216052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.215561    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316475    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsq9\" (UniqueName: \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316524    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316562    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316870 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316614    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316870 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316669    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.316870 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.316740    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.317085 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.317062    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.317162 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.317111    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.317227 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.317202    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.319109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.319086    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.319272 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.319256    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.375887 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.375839    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsq9\" (UniqueName: \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\") pod \"premium-w6-35a187-kserve-67d4df985c-bl9jx\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") " pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.378285 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.378258    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:09.765629 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:09.765593    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 30 23:55:09.769020 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:55:09.768989    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8d8639e_c06b_49e2_82f0_166c0308f907.slice/crio-96e0779bba69783c81762661f4140776f9cdd4a2af616d642ebb9cd5145a1c57 WatchSource:0}: Error finding container 96e0779bba69783c81762661f4140776f9cdd4a2af616d642ebb9cd5145a1c57: Status 404 returned error can't find the container with id 96e0779bba69783c81762661f4140776f9cdd4a2af616d642ebb9cd5145a1c57
Oct 30 23:55:10.670260 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:10.670220    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" event={"ID":"e8d8639e-c06b-49e2-82f0-166c0308f907","Type":"ContainerStarted","Data":"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"}
Oct 30 23:55:10.670681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:10.670273    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" event={"ID":"e8d8639e-c06b-49e2-82f0-166c0308f907","Type":"ContainerStarted","Data":"96e0779bba69783c81762661f4140776f9cdd4a2af616d642ebb9cd5145a1c57"}
Oct 30 23:55:10.782756 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:10.782701    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podStartSLOduration=1.782684684 podStartE2EDuration="1.782684684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:10.781071503 +0000 UTC m=+763.128055398" watchObservedRunningTime="2026-10-30 23:55:10.782684684 +0000 UTC m=+763.129668583"
Oct 30 23:55:11.378970 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:11.378926    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:16.970412 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:55:16.970368    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:55:19.379485 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:19.379444    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:19.392221 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:19.392197    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:19.733204 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:19.733117    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 30 23:55:39.980135 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:39.980096    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:55:46.074515 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.074473    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 30 23:55:46.078892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.078867    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.176002 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.175970    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"openshift-service-ca.crt\""
Oct 30 23:55:46.176002 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.175971    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"kube-root-ca.crt\""
Oct 30 23:55:46.176241 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.176120    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"premium-w0-46b586-kserve-self-signed-certs\""
Oct 30 23:55:46.176241 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.176176    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"default-dockercfg-fst44\""
Oct 30 23:55:46.178127 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.178102    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 30 23:55:46.221861 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.221831    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzn5d\" (UniqueName: \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.222053 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.221882    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.222124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.222068    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.222124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.222105    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.222281 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.222258    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.222346 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.222298    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323263    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzn5d\" (UniqueName: \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323356    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323440    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323463    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323712 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323553    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323712 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323582    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.323985 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.323942    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.324136 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.324005    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.324136 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.324066    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.325882 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.325817    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.326188 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.326171    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.399101 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.399063    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzn5d\" (UniqueName: \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\") pod \"premium-w0-46b586-kserve-66cd484bb9-6wqrp\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") " pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.691444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:46.691363    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:46.969939 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:55:46.969849    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:55:47.060716 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:47.060665    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 30 23:55:47.065139 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:55:47.065111    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7d6d862_8f30_4ca5_a85e_f78b41912730.slice/crio-c168f1a97f67d3715477a69a64dd7fc889a4ab6e5e58451e242c5285a0f58703 WatchSource:0}: Error finding container c168f1a97f67d3715477a69a64dd7fc889a4ab6e5e58451e242c5285a0f58703: Status 404 returned error can't find the container with id c168f1a97f67d3715477a69a64dd7fc889a4ab6e5e58451e242c5285a0f58703
Oct 30 23:55:47.882319 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:47.882282    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" event={"ID":"d7d6d862-8f30-4ca5-a85e-f78b41912730","Type":"ContainerStarted","Data":"56e1cbf78f97da54494365625cd8593f9519125600ced31fc1b2e99857a98e35"}
Oct 30 23:55:47.882798 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:47.882330    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" event={"ID":"d7d6d862-8f30-4ca5-a85e-f78b41912730","Type":"ContainerStarted","Data":"c168f1a97f67d3715477a69a64dd7fc889a4ab6e5e58451e242c5285a0f58703"}
Oct 30 23:55:48.002685 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:48.002626    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podStartSLOduration=2.002610808 podStartE2EDuration="2.002610808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:48.002200845 +0000 UTC m=+800.349184744" watchObservedRunningTime="2026-10-30 23:55:48.002610808 +0000 UTC m=+800.349594706"
Oct 30 23:55:48.692496 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:48.692459    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:56.692019 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:56.691980    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:56.705106 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:56.705076    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:55:56.942575 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:55:56.942493    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 30 23:56:00.157185 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:00.157141    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:56:00.157618 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:00.157386    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy" containerID="cri-o://ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813" gracePeriod=30
Oct 30 23:56:00.185340 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:00.185305    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:56:00.185584 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:00.185542    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy" containerID="cri-o://0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5" gracePeriod=30
Oct 30 23:56:05.480364 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.480329    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:56:05.501994 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.501967    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:56:05.550299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550261    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550334    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550391    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550430    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550470    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550516    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550612    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550651    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550707    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.550929 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.550744    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\") pod \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\" (UID: \"46c63bf8-0ff8-47b7-87d7-ffa7a903be86\") "
Oct 30 23:56:05.551466 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.551206    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.551466 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.551228    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.551466 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.551381    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.551753 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.551721    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:05.551965 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.551942    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.553193 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.553166    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.553325 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.553298    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:56:05.553455 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.553398    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:05.553754 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.553735    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c" pod "46c63bf8-0ff8-47b7-87d7-ffa7a903be86" (UID: "46c63bf8-0ff8-47b7-87d7-ffa7a903be86"). InnerVolumeSpecName "kube-api-access-zjq8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:05.651784 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.651748    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.651784 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.651801    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652059 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.651921    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652059 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.651986    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652059 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652002    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652059 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652041    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652192 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652072    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652192 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652091    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652192 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652111    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\") pod \"46cb77cf-46c7-47dc-8561-1d3b01804105\" (UID: \"46cb77cf-46c7-47dc-8561-1d3b01804105\") "
Oct 30 23:56:05.652490 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652468    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652656 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652636    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjq8c\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-kube-api-access-zjq8c\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652659    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652673    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652688    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652703    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652717    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.652741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652732    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c63bf8-0ff8-47b7-87d7-ffa7a903be86-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.653091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652592    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.653091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652735    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:05.653091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652692    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.653091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.652914    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.653316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.653093    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.654844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.654820    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:05.655024 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.655008    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:56:05.655177 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.655162    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:05.655226 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.655208    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km" pod "46cb77cf-46c7-47dc-8561-1d3b01804105" (UID: "46cb77cf-46c7-47dc-8561-1d3b01804105"). InnerVolumeSpecName "kube-api-access-wb4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:05.753532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753486    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753531    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753545    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753558    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753570    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753582    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753594    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cb77cf-46c7-47dc-8561-1d3b01804105-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753606    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cb77cf-46c7-47dc-8561-1d3b01804105-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.753739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.753619    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4km\" (UniqueName: \"kubernetes.io/projected/46cb77cf-46c7-47dc-8561-1d3b01804105-kube-api-access-wb4km\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:56:05.984485 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.984445    6033 generic.go:356] "Generic (PLEG): container finished" podID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerID="ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813" exitCode=0
Oct 30 23:56:05.984690 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.984515    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"
Oct 30 23:56:05.984690 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.984525    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" event={"ID":"46c63bf8-0ff8-47b7-87d7-ffa7a903be86","Type":"ContainerDied","Data":"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"}
Oct 30 23:56:05.984690 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.984592    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm" event={"ID":"46c63bf8-0ff8-47b7-87d7-ffa7a903be86","Type":"ContainerDied","Data":"fa9699b065dc25029a402a3247e056d5d0edb946126909edca81a96b7b3c74e5"}
Oct 30 23:56:05.984690 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.984623    6033 scope.go:122] "RemoveContainer" containerID="ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"
Oct 30 23:56:05.986241 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.986199    6033 generic.go:356] "Generic (PLEG): container finished" podID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerID="0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5" exitCode=0
Oct 30 23:56:05.986475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.986269    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"
Oct 30 23:56:05.986598 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.986287    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" event={"ID":"46cb77cf-46c7-47dc-8561-1d3b01804105","Type":"ContainerDied","Data":"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"}
Oct 30 23:56:05.986656 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.986600    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97" event={"ID":"46cb77cf-46c7-47dc-8561-1d3b01804105","Type":"ContainerDied","Data":"98dc72817ed62c7e98acad95e656136c98cf9261b4941e2f4cfb95a08f42bf29"}
Oct 30 23:56:05.995308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.995286    6033 scope.go:122] "RemoveContainer" containerID="ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"
Oct 30 23:56:05.995654 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:56:05.995633    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813\": container with ID starting with ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813 not found: ID does not exist" containerID="ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"
Oct 30 23:56:05.995722 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.995662    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813"} err="failed to get container status \"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813\": rpc error: code = NotFound desc = could not find container \"ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813\": container with ID starting with ca13a35ba40af1ae92ebedda8724472a1e68f41de7953b3374919288549ff813 not found: ID does not exist"
Oct 30 23:56:05.995722 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:05.995683    6033 scope.go:122] "RemoveContainer" containerID="0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"
Oct 30 23:56:06.005025 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.004998    6033 scope.go:122] "RemoveContainer" containerID="0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"
Oct 30 23:56:06.005403 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:56:06.005384    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5\": container with ID starting with 0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5 not found: ID does not exist" containerID="0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"
Oct 30 23:56:06.005477 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.005410    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5"} err="failed to get container status \"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5\": rpc error: code = NotFound desc = could not find container \"0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5\": container with ID starting with 0a716414f2f9401c2e44f8186a9228735828c386f8aeb305da9872ab54a14dc5 not found: ID does not exist"
Oct 30 23:56:06.143097 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.143009    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:56:06.156617 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.156585    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-mkvqm"]
Oct 30 23:56:06.260650 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.260562    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:56:06.268230 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:06.268193    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-x5f97"]
Oct 30 23:56:07.839815 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:07.839780    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" path="/var/lib/kubelet/pods/46c63bf8-0ff8-47b7-87d7-ffa7a903be86/volumes"
Oct 30 23:56:07.840447 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:07.840426    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" path="/var/lib/kubelet/pods/46cb77cf-46c7-47dc-8561-1d3b01804105/volumes"
Oct 30 23:56:16.970013 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:56:16.969977    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:56:17.503537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:17.503504    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:56:42.275126 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275089    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 30 23:56:42.275665 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275646    6033 cpu_manager.go:414] "containerMap: removing container" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy"
Oct 30 23:56:42.275718 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275666    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy"
Oct 30 23:56:42.275718 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275700    6033 cpu_manager.go:414] "containerMap: removing container" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy"
Oct 30 23:56:42.275718 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275708    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy"
Oct 30 23:56:42.275859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275800    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cb77cf-46c7-47dc-8561-1d3b01804105" containerName="istio-proxy"
Oct 30 23:56:42.275859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.275811    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c63bf8-0ff8-47b7-87d7-ffa7a903be86" containerName="istio-proxy"
Oct 30 23:56:42.279359 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.279334    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.330975 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.330940    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-27e82b-openshift-default-dockercfg-2pkcf\""
Oct 30 23:56:42.379461 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.379412    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 30 23:56:42.452482 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452444    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452505    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85dz\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452602    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452653    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452781 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452691    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452814 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452786    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452847 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452820    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452885 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452867    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.452954 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.452932    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554026 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.553925    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554026 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554016    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554114    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85dz\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554163    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554189    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554250    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554303    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554343    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554411    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554413    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554519    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554583    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554826 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554810    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.554989 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.554967    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.556681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.556652    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.556934 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.556913    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.613982 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.613948    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.614146 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.614006    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85dz\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:42.893653 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:42.893559    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:43.322223 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:43.322198    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 30 23:56:43.323929 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:56:43.323899    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30985f81_128c_40db_8c21_68d9fb835c35.slice/crio-18447e3abe385eb6797f38ecc19b56c9bd6f50248f2b6c9fc8bbbc7a2389312e WatchSource:0}: Error finding container 18447e3abe385eb6797f38ecc19b56c9bd6f50248f2b6c9fc8bbbc7a2389312e: Status 404 returned error can't find the container with id 18447e3abe385eb6797f38ecc19b56c9bd6f50248f2b6c9fc8bbbc7a2389312e
Oct 30 23:56:43.326419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:43.326383    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:43.326530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:43.326465    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:43.326530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:43.326508    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:44.210265 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.210208    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" event={"ID":"30985f81-128c-40db-8c21-68d9fb835c35","Type":"ContainerStarted","Data":"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"}
Oct 30 23:56:44.210605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.210582    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" event={"ID":"30985f81-128c-40db-8c21-68d9fb835c35","Type":"ContainerStarted","Data":"18447e3abe385eb6797f38ecc19b56c9bd6f50248f2b6c9fc8bbbc7a2389312e"}
Oct 30 23:56:44.376805 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.376748    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podStartSLOduration=2.376731919 podStartE2EDuration="2.376731919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:44.37501206 +0000 UTC m=+856.721995958" watchObservedRunningTime="2026-10-30 23:56:44.376731919 +0000 UTC m=+856.723715873"
Oct 30 23:56:44.900435 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.900395    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:44.902879 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.902847    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:44.903174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:44.903141    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:45.906082 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:45.900439    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:45.906082 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:45.900503    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:46.898452 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:46.898414    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:46.898827 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:46.898799    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:46.969914 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:56:46.969874    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:56:47.906244 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:47.906203    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:47.906466 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:47.906299    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:48.902052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:48.901524    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:48.902052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:48.901584    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:49.894779 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:49.894727    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 23:56:49.895001 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:49.894788    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 23:56:50.899024 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:50.898992    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:50.899487 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:50.899290    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:50.899940 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:50.899919    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 30 23:56:57.067128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.067092    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 30 23:56:57.071400 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.071376    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.121137 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.121105    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-986d37-openshift-default-dockercfg-rm9vh\""
Oct 30 23:56:57.139357 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139319    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139515 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139429    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139515 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139463    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5h9\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139515 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139509    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139685 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139668    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139745 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139729    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139809 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139796    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139825    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.139846 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.139843    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.170259 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.170221    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 30 23:56:57.240950 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.240904    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.240950 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.240946    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5h9\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.240989    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241060    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241098    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241122    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241151    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241178    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241238 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241235    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241356    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241605 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241413    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241715 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241647    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241773 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241705    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.241944 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.241919    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.243517 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.243491    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.243704 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.243681    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.307581 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.307506    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.309450 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.309429    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5h9\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.385230 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.385196    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:57.804198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.804163    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 30 23:56:57.838296 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:56:57.838267    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558c7410_471f_42f3_b60a_e214a76117ee.slice/crio-6b5b1501755402a102473290a2b88c0c29de55842bf333db84144724eb57d352 WatchSource:0}: Error finding container 6b5b1501755402a102473290a2b88c0c29de55842bf333db84144724eb57d352: Status 404 returned error can't find the container with id 6b5b1501755402a102473290a2b88c0c29de55842bf333db84144724eb57d352
Oct 30 23:56:57.840678 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.840643    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:57.840739 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.840714    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:57.840775 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:57.840746    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:56:58.291325 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.291274    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" event={"ID":"558c7410-471f-42f3-b60a-e214a76117ee","Type":"ContainerStarted","Data":"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"}
Oct 30 23:56:58.291919 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.291343    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" event={"ID":"558c7410-471f-42f3-b60a-e214a76117ee","Type":"ContainerStarted","Data":"6b5b1501755402a102473290a2b88c0c29de55842bf333db84144724eb57d352"}
Oct 30 23:56:58.396161 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.396078    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:56:58.396441 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.396419    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:56:58.396593 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.396570    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:56:58.409995 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.409851    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podStartSLOduration=1.409834274 podStartE2EDuration="1.409834274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:58.408489736 +0000 UTC m=+870.755473669" watchObservedRunningTime="2026-10-30 23:56:58.409834274 +0000 UTC m=+870.756818175"
Oct 30 23:56:58.445580 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.445545    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"]
Oct 30 23:56:58.449518 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.449491    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.505863 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.505820    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"]
Oct 30 23:56:58.508044 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.506410    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:58.508252 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.507094    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 30 23:56:58.557803 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.557719    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.557803 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.557785    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfwp\" (UniqueName: \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.557990 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.557919    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.659674 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.659634    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.659884 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.659695    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfwp\" (UniqueName: \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.659884 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.659727    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.660443 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.660420    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.662042 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.662010    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.721297 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.721262    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfwp\" (UniqueName: \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\") pod \"maas-api-revoke-sub-0f450a80-qqq2d\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:58.764009 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:58.763980    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:56:59.169527 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:59.169492    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"]
Oct 30 23:56:59.172701 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:56:59.172652    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dde4d4_0afd_4fa9_8627_f4d9efb386d7.slice/crio-a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5 WatchSource:0}: Error finding container a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5: Status 404 returned error can't find the container with id a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5
Oct 30 23:56:59.174871 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:59.174853    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:56:59.312592 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:59.312557    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d" event={"ID":"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7","Type":"ContainerStarted","Data":"a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5"}
Oct 30 23:56:59.408055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:59.406528    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:56:59.408055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:56:59.406593    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:57:00.404054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:00.396443    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:57:00.404054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:00.396505    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:57:01.405187 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:01.405143    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:57:01.405799 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:01.405762    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:57:02.407548 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:02.407113    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:57:02.407548 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:02.407175    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:57:03.408203 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:03.408158    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 23:57:03.408590 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:03.408244    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 23:57:03.902637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:03.902601    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:57:04.699112 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:04.698500    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:57:04.699112 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:04.698910    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:57:04.699982 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:04.699960    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 30 23:57:05.360557 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:05.360514    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d" event={"ID":"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7","Type":"ContainerStarted","Data":"450343b38b9dba1dfffc86afc7ff4d5ea21c3c3ee8d6ce0d72b7f629e5f440f9"}
Oct 30 23:57:05.472360 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:05.472308    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d" podStartSLOduration=2.264249425 podStartE2EDuration="7.472294368s" totalImagesPullingTime="5.208044943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:05.470787257 +0000 UTC m=+877.817771159" watchObservedRunningTime="2026-10-30 23:57:05.472294368 +0000 UTC m=+877.819278266"
Oct 30 23:57:06.366720 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:06.366685    6033 generic.go:356] "Generic (PLEG): container finished" podID="f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" containerID="450343b38b9dba1dfffc86afc7ff4d5ea21c3c3ee8d6ce0d72b7f629e5f440f9" exitCode=0
Oct 30 23:57:06.367198 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:06.366757    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d" event={"ID":"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7","Type":"ContainerDied","Data":"450343b38b9dba1dfffc86afc7ff4d5ea21c3c3ee8d6ce0d72b7f629e5f440f9"}
Oct 30 23:57:07.510329 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.510295    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:57:07.700276 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.700200    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\" (UniqueName: \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\") pod \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") "
Oct 30 23:57:07.700424 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.700378    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\") pod \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") "
Oct 30 23:57:07.700424 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.700401    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\") pod \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\" (UID: \"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7\") "
Oct 30 23:57:07.700826 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.700797    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca" pod "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" (UID: "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:07.702537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.702506    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca" pod "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" (UID: "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:07.702537 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.702523    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp" pod "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" (UID: "f2dde4d4-0afd-4fa9-8627-f4d9efb386d7"). InnerVolumeSpecName "kube-api-access-4vfwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:07.801597 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.801546    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:07.801597 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.801591    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:07.801597 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:07.801601    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfwp\" (UniqueName: \"kubernetes.io/projected/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7-kube-api-access-4vfwp\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:08.379816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:08.379782    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d" event={"ID":"f2dde4d4-0afd-4fa9-8627-f4d9efb386d7","Type":"ContainerDied","Data":"a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5"}
Oct 30 23:57:08.379816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:08.379815    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"
Oct 30 23:57:08.380048 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:08.379831    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52a048d60e4c3917e8161756a29ed4f8eb1378d3c072c0dc9298b56597d9bf5"
Oct 30 23:57:15.454159 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.454125    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"]
Oct 30 23:57:15.454681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.454667    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" containerName="revoke-keys"
Oct 30 23:57:15.454751 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.454681    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" containerName="revoke-keys"
Oct 30 23:57:15.454789 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.454772    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" containerName="revoke-keys"
Oct 30 23:57:15.458237 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.458218    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.504824 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.504786    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbv5f\" (UniqueName: \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.504972 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.504917    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.504972 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.504946    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.512305 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.512277    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"]
Oct 30 23:57:15.555962 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.555929    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:15.556231 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.556211    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 30 23:57:15.606311 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.606271    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.606481 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.606317    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.606481 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.606418    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbv5f\" (UniqueName: \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.606943 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.606921    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.608705 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.608683    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.664851 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.664814    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbv5f\" (UniqueName: \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\") pod \"maas-api-revoke-sub-dbbc0b2b-rzqqx\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:15.770807 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:15.770769    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:16.141558 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:16.141531    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"]
Oct 30 23:57:16.143737 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:57:16.143710    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc493130_2a56_4a89_bf75_49eb4ef11fea.slice/crio-119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b WatchSource:0}: Error finding container 119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b: Status 404 returned error can't find the container with id 119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b
Oct 30 23:57:16.424642 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:16.424554    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx" event={"ID":"fc493130-2a56-4a89-bf75-49eb4ef11fea","Type":"ContainerStarted","Data":"119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b"}
Oct 30 23:57:16.969389 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:57:16.969350    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:57:17.432680 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:17.432645    6033 generic.go:356] "Generic (PLEG): container finished" podID="fc493130-2a56-4a89-bf75-49eb4ef11fea" containerID="d98936319ed0d4157b2f7bcaf333a6c532da8d916b026da21516a2ac4440b349" exitCode=0
Oct 30 23:57:17.432876 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:17.432728    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx" event={"ID":"fc493130-2a56-4a89-bf75-49eb4ef11fea","Type":"ContainerDied","Data":"d98936319ed0d4157b2f7bcaf333a6c532da8d916b026da21516a2ac4440b349"}
Oct 30 23:57:18.586867 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.586834    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:18.641475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.641444    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\" (UniqueName: \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\") pod \"fc493130-2a56-4a89-bf75-49eb4ef11fea\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") "
Oct 30 23:57:18.641475 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.641484    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\") pod \"fc493130-2a56-4a89-bf75-49eb4ef11fea\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") "
Oct 30 23:57:18.641745 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.641679    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\") pod \"fc493130-2a56-4a89-bf75-49eb4ef11fea\" (UID: \"fc493130-2a56-4a89-bf75-49eb4ef11fea\") "
Oct 30 23:57:18.642084 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.642057    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca" pod "fc493130-2a56-4a89-bf75-49eb4ef11fea" (UID: "fc493130-2a56-4a89-bf75-49eb4ef11fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:18.643779 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.643753    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f" pod "fc493130-2a56-4a89-bf75-49eb4ef11fea" (UID: "fc493130-2a56-4a89-bf75-49eb4ef11fea"). InnerVolumeSpecName "kube-api-access-wbv5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:18.643876 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.643786    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca" pod "fc493130-2a56-4a89-bf75-49eb4ef11fea" (UID: "fc493130-2a56-4a89-bf75-49eb4ef11fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:18.743262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.743184    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:18.743262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.743213    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbv5f\" (UniqueName: \"kubernetes.io/projected/fc493130-2a56-4a89-bf75-49eb4ef11fea-kube-api-access-wbv5f\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:18.743262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:18.743225    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc493130-2a56-4a89-bf75-49eb4ef11fea-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 23:57:19.446560 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:19.446526    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"
Oct 30 23:57:19.446740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:19.446559    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx" event={"ID":"fc493130-2a56-4a89-bf75-49eb4ef11fea","Type":"ContainerDied","Data":"119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b"}
Oct 30 23:57:19.446740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:19.446636    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119ad3baf8e5e355baa65f120890af58b746ef71a8ae844abe777dfd35f4872b"
Oct 30 23:57:30.153315 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.153275    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 30 23:57:30.154175 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.154152    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fc493130-2a56-4a89-bf75-49eb4ef11fea" containerName="revoke-keys"
Oct 30 23:57:30.154235 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.154178    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc493130-2a56-4a89-bf75-49eb4ef11fea" containerName="revoke-keys"
Oct 30 23:57:30.154350 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.154336    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc493130-2a56-4a89-bf75-49eb4ef11fea" containerName="revoke-keys"
Oct 30 23:57:30.159552 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.159528    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.198731 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.198698    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 30 23:57:30.202807 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.202783    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.243558 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.243513    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\""
Oct 30 23:57:30.247133 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.247102    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 30 23:57:30.266009 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.265975    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 30 23:57:30.277740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.277707    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.277740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.277743    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.277856    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.277977    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.278080    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278273 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.278113    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278273 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.278141    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thtc\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278273 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.278188    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.278435 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.278300    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379267    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379317    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379563 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379363    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379563 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379385    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379563 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379476    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379563 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379511    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379587    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379693    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379715    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379745    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379824    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.379888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379849    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379877    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.379888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.379881    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.380144 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380086    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380144 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380125    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380161    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380167    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchcc\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.380236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380225    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thtc\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380382 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380252    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.380382 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380285    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380382 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380361    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.380727 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.380699    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.382076 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.382051    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.382659 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.382641    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.438304 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.438219    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.438451 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.438301    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thtc\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-q6mtd\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.473212 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.473169    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:30.481181 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481154    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481209    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchcc\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481301 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481248    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481318    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481378    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481419 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481402    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481454    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481514    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481586 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481573    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481741 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481583    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481797 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481734    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.481933 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.481908    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.482045 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.482010    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.482202 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.482181    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.483862 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.483838    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.484296 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.484278    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.542001 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.541959    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchcc\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.542495 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.542466    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-cxzjj\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.816756 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.816719    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:30.895151 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.895114    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 30 23:57:30.899017 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:57:30.898987    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98204b8_0c6c_49a3_acc4_5d90832930e4.slice/crio-58feece8513115befa0d7db85feb469c2c5584a777a7e28730a672bc9a2f1546 WatchSource:0}: Error finding container 58feece8513115befa0d7db85feb469c2c5584a777a7e28730a672bc9a2f1546: Status 404 returned error can't find the container with id 58feece8513115befa0d7db85feb469c2c5584a777a7e28730a672bc9a2f1546
Oct 30 23:57:30.902413 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.902372    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:30.902533 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.902450    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:30.902533 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:30.902484    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:31.190598 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.190572    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 30 23:57:31.192762 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:57:31.192738    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd905008c_4172_400f_bbdf_64473060620f.slice/crio-81cbbb08e46db06aa6b67b974bac925f4aa859dc3762ca6a8776801e006d9d48 WatchSource:0}: Error finding container 81cbbb08e46db06aa6b67b974bac925f4aa859dc3762ca6a8776801e006d9d48: Status 404 returned error can't find the container with id 81cbbb08e46db06aa6b67b974bac925f4aa859dc3762ca6a8776801e006d9d48
Oct 30 23:57:31.195794 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.195762    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:31.195873 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.195826    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:31.195873 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.195855    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:31.515843 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.515795    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" event={"ID":"f98204b8-0c6c-49a3-acc4-5d90832930e4","Type":"ContainerStarted","Data":"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"}
Oct 30 23:57:31.516082 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.515877    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" event={"ID":"f98204b8-0c6c-49a3-acc4-5d90832930e4","Type":"ContainerStarted","Data":"58feece8513115befa0d7db85feb469c2c5584a777a7e28730a672bc9a2f1546"}
Oct 30 23:57:31.517888 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.517860    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" event={"ID":"d905008c-4172-400f-bbdf-64473060620f","Type":"ContainerStarted","Data":"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"}
Oct 30 23:57:31.518023 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.517916    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" event={"ID":"d905008c-4172-400f-bbdf-64473060620f","Type":"ContainerStarted","Data":"81cbbb08e46db06aa6b67b974bac925f4aa859dc3762ca6a8776801e006d9d48"}
Oct 30 23:57:31.633017 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.632966    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podStartSLOduration=1.632952706 podStartE2EDuration="1.632952706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:31.630460567 +0000 UTC m=+903.977444465" watchObservedRunningTime="2026-10-30 23:57:31.632952706 +0000 UTC m=+903.979936652"
Oct 30 23:57:31.745470 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.745417    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podStartSLOduration=1.745401091 podStartE2EDuration="1.745401091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:31.743820412 +0000 UTC m=+904.090804315" watchObservedRunningTime="2026-10-30 23:57:31.745401091 +0000 UTC m=+904.092384989"
Oct 30 23:57:31.817416 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:31.817336    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:32.473628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:32.473597    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:32.475381 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:32.475348    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:32.475534 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:32.475403    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:32.831056 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:32.819850    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:32.831056 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:32.819919    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:33.474504 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:33.474466    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:33.474906 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:33.474526    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:33.818057 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:33.817485    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:33.818057 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:33.817562    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:34.474629 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:34.474593    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:34.475075 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:34.474654    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:34.818505 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:34.818464    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:34.818810 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:34.818779    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:35.474072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:35.473816    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:35.474072 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:35.473872    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:35.821055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:35.820136    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:35.821055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:35.820202    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:36.473993 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:36.473952    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:36.474206 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:36.474013    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:36.819063 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:36.818623    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:36.819063 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:36.818689    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:37.474636 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:37.474599    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:37.475069 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:37.474665    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:37.817334 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:37.817302    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:37.817524 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:37.817354    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:38.474452 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:38.474411    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-q6mtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:57:38.474653 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:38.474473    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 23:57:38.818038 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:38.817996    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-cxzjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 23:57:38.818233 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:38.818085    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 23:57:40.031096 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.031066    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:40.031509 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.031401    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:40.032454 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.032430    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 30 23:57:40.259110 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.257878    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:40.259110 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.258369    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:40.259803 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:40.259785    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 30 23:57:45.157241 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.157201    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 30 23:57:45.160763 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.160736    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.272942 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.272906    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 30 23:57:45.281584 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281551    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.281762 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281594    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.281762 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281722    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.281762 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281761    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.281947 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281791    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.281947 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281848    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.282089 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281941    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.282089 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.281976    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtxn\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.282089 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.282050    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383552 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383516    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383722 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383558    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383766 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383714    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383818 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383776    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtxn\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383866 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383825    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383921 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383869    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.383921 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383896    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384019 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.383957    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384054    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384119 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384082    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384200 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384116    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384270 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384251    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384343 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384322    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.384737 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.384718    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.386338 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.386314    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.386478 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.386459    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.443306 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.443235    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.443458 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.443326    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtxn\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\") pod \"maas-default-gateway-openshift-default-8f6f87f59-m59d5\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.477316 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.477279    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:45.899651 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.899624    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 30 23:57:45.901233 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:57:45.901201    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09448aa_e522_4109_8e39_a7e295f4237f.slice/crio-702a7f55dc9ecb6e50abbd83fa92d88ec139363b1587e4130c368f2483465e61 WatchSource:0}: Error finding container 702a7f55dc9ecb6e50abbd83fa92d88ec139363b1587e4130c368f2483465e61: Status 404 returned error can't find the container with id 702a7f55dc9ecb6e50abbd83fa92d88ec139363b1587e4130c368f2483465e61
Oct 30 23:57:45.903954 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.903911    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:45.904077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.903988    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:45.904077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:45.904015    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:57:46.623777 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:46.623739    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" event={"ID":"e09448aa-e522-4109-8e39-a7e295f4237f","Type":"ContainerStarted","Data":"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"}
Oct 30 23:57:46.624328 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:46.623808    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" event={"ID":"e09448aa-e522-4109-8e39-a7e295f4237f","Type":"ContainerStarted","Data":"702a7f55dc9ecb6e50abbd83fa92d88ec139363b1587e4130c368f2483465e61"}
Oct 30 23:57:46.970518 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:57:46.970428    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:57:47.478137 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:47.478102    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:47.479837 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:47.479806    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:47.479978 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:47.479855    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:48.478152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:48.478115    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:48.478580 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:48.478170    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:49.477813 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:49.477772    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:49.478075 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:49.477835    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:50.478716 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:50.478678    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:50.479187 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:50.478732    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:51.478278 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:51.478240    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:51.478457 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:51.478306    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:52.478178 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:52.478130    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:52.478728 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:52.478193    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:53.002516 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:53.002452    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podStartSLOduration=8.002434297 podStartE2EDuration="8.002434297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:46.812713316 +0000 UTC m=+919.159697215" watchObservedRunningTime="2026-10-30 23:57:53.002434297 +0000 UTC m=+925.349418195"
Oct 30 23:57:53.004891 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:53.004858    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:57:53.478216 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:53.478122    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-m59d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 23:57:53.478216 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:53.478183    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 23:57:54.985354 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:54.985317    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:54.993054 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:54.989331    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:57:54.998348 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:57:54.998319    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 30 23:58:12.770167 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.770130    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 30 23:58:12.775424 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.775394    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.830107    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.831370    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.831615    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.831700    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.831806    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjsn\" (UniqueName: \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.831926    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.834079 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.832075    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.873714 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.873684    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"openshift-service-ca.crt\""
Oct 30 23:58:12.873714 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.873714    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"kube-root-ca.crt\""
Oct 30 23:58:12.933585 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933536    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.933791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933613    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.933791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933658    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.933791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933689    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.933791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933736    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjsn\" (UniqueName: \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.933791 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.933790    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.934139 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.934106    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.934262 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.934236    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.934507 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.934469    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.936643 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.936603    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:12.958388 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.958357    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"distinct-w4-34f1ce-kserve-self-signed-certs\""
Oct 30 23:58:12.958388 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.958363    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-34f1ce\"/\"default-dockercfg-rrmnx\""
Oct 30 23:58:12.967166 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:12.967140    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:13.012221 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.012182    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjsn\" (UniqueName: \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\") pod \"distinct-w4-34f1ce-kserve-85744459ff-7sqnh\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") " pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:13.093203 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.093103    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:13.553564 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.553536    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 30 23:58:13.556002 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:58:13.555972    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ce9f007_07d1_4172_819b_27b14cecc0a7.slice/crio-d6591d70ef6323bef796c27c5b6023388ec695e835bbce1f19766d6ef786c31c WatchSource:0}: Error finding container d6591d70ef6323bef796c27c5b6023388ec695e835bbce1f19766d6ef786c31c: Status 404 returned error can't find the container with id d6591d70ef6323bef796c27c5b6023388ec695e835bbce1f19766d6ef786c31c
Oct 30 23:58:13.767318 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.767282    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 30 23:58:13.770559 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.770519    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.808494 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.808402    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" event={"ID":"8ce9f007-07d1-4172-819b-27b14cecc0a7","Type":"ContainerStarted","Data":"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"}
Oct 30 23:58:13.808748 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.808725    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" event={"ID":"8ce9f007-07d1-4172-819b-27b14cecc0a7","Type":"ContainerStarted","Data":"d6591d70ef6323bef796c27c5b6023388ec695e835bbce1f19766d6ef786c31c"}
Oct 30 23:58:13.864258 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.864176    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 30 23:58:13.896939 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.896889    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897150 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897114    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897243 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897213    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897312 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897300    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897437 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897357    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wdh\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897508 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897436    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897508 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897486    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897694 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897551    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.897750 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.897710    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998524 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998487    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998719 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998553    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wdh\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998798 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998737    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998854 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998813    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998854 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998842    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.998958 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998932    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999014 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.998966    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999118 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999018    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999118 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999097    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999117    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999222 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999182    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999392 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999372    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999435 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999390    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:13.999668 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:13.999648    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.001076 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.001050    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.001637 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.001614    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.006952 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.006908    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podStartSLOduration=2.006897033 podStartE2EDuration="2.006897033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:14.005115255 +0000 UTC m=+946.352099153" watchObservedRunningTime="2026-10-30 23:58:14.006897033 +0000 UTC m=+946.353880931"
Oct 30 23:58:14.056856 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.056828    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.057005 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.056894    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wdh\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.093878 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.093788    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:14.100290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.100263    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:14.534540 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.534514    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 30 23:58:14.536967 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:58:14.536932    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecce1a5a_d4ce_42ea_b2e8_53c43ea33d7f.slice/crio-8ccd767f3a699fa20b72a9a23332253478e38de381e045d409e1a6a32a8648a2 WatchSource:0}: Error finding container 8ccd767f3a699fa20b72a9a23332253478e38de381e045d409e1a6a32a8648a2: Status 404 returned error can't find the container with id 8ccd767f3a699fa20b72a9a23332253478e38de381e045d409e1a6a32a8648a2
Oct 30 23:58:14.539484 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.539447    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:14.539544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.539517    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:14.539588 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.539545    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:14.816867 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.816730    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" event={"ID":"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f","Type":"ContainerStarted","Data":"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"}
Oct 30 23:58:14.816867 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.816796    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" event={"ID":"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f","Type":"ContainerStarted","Data":"8ccd767f3a699fa20b72a9a23332253478e38de381e045d409e1a6a32a8648a2"}
Oct 30 23:58:14.931214 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:14.931152    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podStartSLOduration=1.931134339 podStartE2EDuration="1.931134339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:14.929591314 +0000 UTC m=+947.276575212" watchObservedRunningTime="2026-10-30 23:58:14.931134339 +0000 UTC m=+947.278118237"
Oct 30 23:58:15.106174 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:15.101379    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:15.107774 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:15.107625    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:15.107774 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:15.107686    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:16.106657 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:16.106205    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:16.106657 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:16.106270    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:16.970282 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:58:16.970248    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:58:17.101513 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:17.101475    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:17.101823 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:17.101795    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:18.106496 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:18.106458    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:18.106918 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:18.106533    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:19.104480 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:19.104442    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:19.104812 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:19.104777    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:20.105052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:20.104517    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:20.105052 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:20.104581    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:21.101727 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:21.101680    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 23:58:21.101885 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:21.101764    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 23:58:22.105267 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:22.105236    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:22.105724 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:22.105537    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:22.106261 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:22.106241    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 30 23:58:23.093730 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:23.093697    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:23.106436 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:23.106411    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:23.881019 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:23.880987    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 30 23:58:24.038416 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:24.038383    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:58:32.252841 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.252802    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 30 23:58:32.256822 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.256801    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326231 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326194    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326420 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326245    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326420 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326371    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326522 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326493    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsj6\" (UniqueName: \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326560 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326526    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.326639 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.326627    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.352539 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.352510    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5c3374\"/\"distinct-w1-5c3374-kserve-self-signed-certs\""
Oct 30 23:58:32.354773 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.354742    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 30 23:58:32.427824 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.427783    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428000 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.427855    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428000 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.427912    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428161 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.427998    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsj6\" (UniqueName: \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428161 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.428044    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428257 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.428170    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428548 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.428516    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.428803 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.428780    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.429010 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.428992    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.430527 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.430495    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.430870 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.430842    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.486544 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.486500    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsj6\" (UniqueName: \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\") pod \"distinct-w1-5c3374-kserve-549997c7c7-nsq72\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") " pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.568607 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.568502    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:32.948084 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:32.948022    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 30 23:58:33.935313 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:33.935274    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" event={"ID":"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b","Type":"ContainerStarted","Data":"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"}
Oct 30 23:58:33.935712 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:33.935323    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" event={"ID":"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b","Type":"ContainerStarted","Data":"0e4832d64394ed03fc4d1b722e15d546559539843022942561825a98ead2d105"}
Oct 30 23:58:34.050511 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:34.050456    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podStartSLOduration=2.050443455 podStartE2EDuration="2.050443455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:34.049903493 +0000 UTC m=+966.396887391" watchObservedRunningTime="2026-10-30 23:58:34.050443455 +0000 UTC m=+966.397427352"
Oct 30 23:58:34.568802 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:34.568760    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:42.530723 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.530671    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 30 23:58:42.534110 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.534085    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552617 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552588    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mwm\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552636    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552740 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552705    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552833 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552811    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552869 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552844    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552956 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552935    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.552991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.552979    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.553082 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.553058    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.553128 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.553101    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.569000 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.568963    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:42.581875 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.581847    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:42.583870 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.583845    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-34f1ce-openshift-default-dockercfg-gfqnh\""
Oct 30 23:58:42.633618 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.633584    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 30 23:58:42.654784 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.654739    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.654991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.654817    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.654991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.654876    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.654991 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.654929    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mwm\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655201 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655172    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655258 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655227    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655255    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655235    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655421 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655375    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655421 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655409    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655523 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655502    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.655827 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.655792    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.656109 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.656084    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.656294 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.656270    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.657878 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.657853    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.658440 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.658421    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.713118 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.713088    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.713256 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.713141    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mwm\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:42.847909 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:42.847821    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:43.005310 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:43.005278    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 30 23:58:43.291340 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:43.291314    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 30 23:58:43.294952 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:58:43.294904    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea30a07_c3e3_45c0_8f8f_36b242237879.slice/crio-939e345c0c7a0334b652d510f3e61e04b000f47c66145ad94015016a4db1c7ff WatchSource:0}: Error finding container 939e345c0c7a0334b652d510f3e61e04b000f47c66145ad94015016a4db1c7ff: Status 404 returned error can't find the container with id 939e345c0c7a0334b652d510f3e61e04b000f47c66145ad94015016a4db1c7ff
Oct 30 23:58:43.297534 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:43.297493    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:43.297599 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:43.297570    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:43.297645 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:43.297601    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:58:44.019168 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.019098    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" event={"ID":"7ea30a07-c3e3-45c0-8f8f-36b242237879","Type":"ContainerStarted","Data":"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"}
Oct 30 23:58:44.019168 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.019164    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" event={"ID":"7ea30a07-c3e3-45c0-8f8f-36b242237879","Type":"ContainerStarted","Data":"939e345c0c7a0334b652d510f3e61e04b000f47c66145ad94015016a4db1c7ff"}
Oct 30 23:58:44.136071 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.135714    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podStartSLOduration=2.135692551 podStartE2EDuration="2.135692551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:44.133790183 +0000 UTC m=+976.480774079" watchObservedRunningTime="2026-10-30 23:58:44.135692551 +0000 UTC m=+976.482676451"
Oct 30 23:58:44.848981 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.848939    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:44.850831 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.850798    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:44.850971 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:44.850851    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:45.848205 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:45.848170    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:45.848595 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:45.848234    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:46.848442 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:46.848402    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:46.848859 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:46.848459    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:46.971359 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:58:46.971317    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:58:47.849091 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:47.849016    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:47.849514 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:47.849112    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:48.848635 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:48.848602    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:48.848819 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:48.848659    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:49.849260 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:49.849220    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 23:58:49.849719 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:49.849285    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 23:58:50.853525 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:50.853489    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:50.853920 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:50.853744    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:50.854584 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:50.854561    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 30 23:58:59.387541 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.387501    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 30 23:58:59.392044 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.392006    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.444047 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.444005    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fe01fa-openshift-default-dockercfg-5mw89\""
Oct 30 23:58:59.473796 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.473763    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.473796 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.473802    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.473993 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.473916    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474063 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.473982    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474063 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.474011    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474131 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.474118    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474177 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.474163    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474212 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.474193    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.474321 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.474307    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcz5\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.492975 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.492947    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 30 23:58:59.575606 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575565    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.575606 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575613    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.575890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575680    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcz5\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.575890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575755    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.575890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575785    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.575890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575874    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575916    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575941    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576152 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.575987    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576315 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.576162    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576372 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.576343    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576372 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.576359    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576478 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.576455    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.576749 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.576728    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.578290 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.578267    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.578402 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.578311    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.633730 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.633698    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.634175 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.634150    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcz5\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:58:59.706286 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:58:59.706182    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:59:00.125980 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:00.125955    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 30 23:59:00.127930 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:59:00.127900    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476bf414_e7cc_48f3_b062_6827959edc9f.slice/crio-982ca16c723ff7763839279f7a8cd2551478816aeaaab7aa0b404f769f7aae00 WatchSource:0}: Error finding container 982ca16c723ff7763839279f7a8cd2551478816aeaaab7aa0b404f769f7aae00: Status 404 returned error can't find the container with id 982ca16c723ff7763839279f7a8cd2551478816aeaaab7aa0b404f769f7aae00
Oct 30 23:59:00.130417 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:00.130382    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:00.130491 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:00.130453    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:00.130491 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:00.130481    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:01.124156 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.124117    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" event={"ID":"476bf414-e7cc-48f3-b062-6827959edc9f","Type":"ContainerStarted","Data":"150055cfa5a6a4b9d7e4866b480002a54f54f8d2830c3cf790392d2e8c952881"}
Oct 30 23:59:01.124621 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.124175    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" event={"ID":"476bf414-e7cc-48f3-b062-6827959edc9f","Type":"ContainerStarted","Data":"982ca16c723ff7763839279f7a8cd2551478816aeaaab7aa0b404f769f7aae00"}
Oct 30 23:59:01.346917 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.346864    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podStartSLOduration=2.346848801 podStartE2EDuration="2.346848801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:01.299535555 +0000 UTC m=+993.646519488" watchObservedRunningTime="2026-10-30 23:59:01.346848801 +0000 UTC m=+993.693832698"
Oct 30 23:59:01.707395 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.707351    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:59:01.709366 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.709335    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:01.709543 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:01.709413    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:02.707747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:02.707695    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:02.708308 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:02.707798    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:03.707664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:03.707619    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:03.707853 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:03.707704    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:04.707489 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:04.707449    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:04.707819 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:04.707783    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:05.707492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:05.707456    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:05.707966 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:05.707935    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:06.261601 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:06.261505    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:59:06.707259 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:06.707178    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:59:06.707259 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:06.707239    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 23:59:07.714053 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:07.713190    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:59:07.714053 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:07.713760    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:59:07.715217 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:07.715165    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 30 23:59:16.970005 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:59:16.969963    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:59:28.946527 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.946493    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 30 23:59:28.950340 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.950315    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999086    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999124 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999129    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999257    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999373 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999307    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999449 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999420    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999482 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999461    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rnf\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999530 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999513    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999608 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999541    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:28.999651 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:28.999632    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.050722 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.050687    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 30 23:59:29.101347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101309    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101347 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101351    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101404    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101432    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101454    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101475    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rnf\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101503    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101539    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101597    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101977 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101864    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.101977 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.101963    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.102110 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.102043    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.102289 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.102270    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.102388 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.102365    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.103951 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.103905    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.104228 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.104198    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.159473 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.159435    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.159654 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.159633    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rnf\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\") pod \"e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") " pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.263662 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.263618    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:29.694676 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.694648    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 30 23:59:29.696218 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:59:29.696186    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392b5af8_5042_4f20_a6dc_8d4ac3a62c81.slice/crio-0c09d045cd9b7c904824b78b7595bf2455ab130dea86fd7bca7db4e79cf9e223 WatchSource:0}: Error finding container 0c09d045cd9b7c904824b78b7595bf2455ab130dea86fd7bca7db4e79cf9e223: Status 404 returned error can't find the container with id 0c09d045cd9b7c904824b78b7595bf2455ab130dea86fd7bca7db4e79cf9e223
Oct 30 23:59:29.699139 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.699098    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:29.699211 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.699195    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:29.699251 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:29.699236    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:30.319468 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:30.319432    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" event={"ID":"392b5af8-5042-4f20-a6dc-8d4ac3a62c81","Type":"ContainerStarted","Data":"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"}
Oct 30 23:59:30.319909 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:30.319501    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" event={"ID":"392b5af8-5042-4f20-a6dc-8d4ac3a62c81","Type":"ContainerStarted","Data":"0c09d045cd9b7c904824b78b7595bf2455ab130dea86fd7bca7db4e79cf9e223"}
Oct 30 23:59:30.485053 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:30.484535    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podStartSLOduration=2.484513734 podStartE2EDuration="2.484513734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:30.481591426 +0000 UTC m=+1022.828575352" watchObservedRunningTime="2026-10-30 23:59:30.484513734 +0000 UTC m=+1022.831497633"
Oct 30 23:59:31.264445 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:31.264405    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:31.266502 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:31.266465    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:31.266663 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:31.266517    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:32.264777 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:32.264743    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:32.265214 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:32.264813    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:33.209500 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.209462    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 30 23:59:33.213272 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.213244    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254001 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.253961    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254345 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.254319    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254492 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.254477    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254652 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.254636    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtl8\" (UniqueName: \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254816 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.254802    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.254919 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.254906    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.264873 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.264834    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:33.265292 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.264916    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:33.311006 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.310963    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 30 23:59:33.312942 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.312906    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-46b586\"/\"distinct-2-w0-46b586-kserve-self-signed-certs\""
Oct 30 23:59:33.356115 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356074    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtl8\" (UniqueName: \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356286 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356162    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356286 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356193    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356409 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356358    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356463 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356449    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356532 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356501    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356693 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356675    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356805 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356786    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.356892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.356867    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.358639 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.358609    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.358916 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.358890    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.434182 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.434140    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtl8\" (UniqueName: \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\") pod \"distinct-2-w0-46b586-kserve-6f7d78987-l7xtj\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") " pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.526628 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.526596    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:33.938768 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:33.938680    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 30 23:59:33.943955 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:59:33.943925    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042b4b34_8fc8_4171_a166_78bac0196e01.slice/crio-1d949b46c47b1673f415e10d4c119ecbbe5a9f1e2f11b995ae187a21539e6b1e WatchSource:0}: Error finding container 1d949b46c47b1673f415e10d4c119ecbbe5a9f1e2f11b995ae187a21539e6b1e: Status 404 returned error can't find the container with id 1d949b46c47b1673f415e10d4c119ecbbe5a9f1e2f11b995ae187a21539e6b1e
Oct 30 23:59:34.264114 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.264086    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:34.264257 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.264139    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:34.347712 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.347677    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" event={"ID":"042b4b34-8fc8-4171-a166-78bac0196e01","Type":"ContainerStarted","Data":"3cb804a49653c6bf9a8a775d105ade9ea366a2189adfa711bbf7268eb3f374c5"}
Oct 30 23:59:34.348141 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.347720    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" event={"ID":"042b4b34-8fc8-4171-a166-78bac0196e01","Type":"ContainerStarted","Data":"1d949b46c47b1673f415e10d4c119ecbbe5a9f1e2f11b995ae187a21539e6b1e"}
Oct 30 23:59:34.467685 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.467627    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podStartSLOduration=1.4676111459999999 podStartE2EDuration="1.467611146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:34.465098791 +0000 UTC m=+1026.812082686" watchObservedRunningTime="2026-10-30 23:59:34.467611146 +0000 UTC m=+1026.814595044"
Oct 30 23:59:34.526885 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:34.526792    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:35.264354 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:35.264319    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:35.264509 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:35.264375    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:36.264327 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:36.264293    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 23:59:36.264711 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:36.264352    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 23:59:37.268528 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:37.268497    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:37.268949 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:37.268812    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:37.269502 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:37.269479    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 30 23:59:43.527703 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.527653    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:43.540701 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.540671    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:43.701829 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.701787    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 30 23:59:43.706099 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.706077    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.799444 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.799339    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 30 23:59:43.806196 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806161    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806361 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806208    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806361 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806239    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806453 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806383    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29z74\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806506 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806486    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806559 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806545    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806655 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806639    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806697 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806687    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.806763 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.806749    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907512 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907478    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907531    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907552    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907579    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907681 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907641    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29z74\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907733    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907785    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.907892 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907884    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907910    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907935    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908077 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.907959    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908318 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.908295    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908377 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.908334    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.908627 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.908606    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.909947 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.909915    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.910422 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.910400    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.967103 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.967073    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29z74\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:43.967267 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:43.967243    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:44.020460 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.020429    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:44.333764 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.333686    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 30 23:59:44.420959 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.420926    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 30 23:59:44.489147 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.489126    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 30 23:59:44.490900 ip-10-0-139-66 kubenswrapper[6033]: W1030 23:59:44.490864    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418c94f0_9f39_4846_b629_808c772500fc.slice/crio-3d91aeaecf0ebefe2981906f746401f6dc99e03b0af6a86a83596bb910258f91 WatchSource:0}: Error finding container 3d91aeaecf0ebefe2981906f746401f6dc99e03b0af6a86a83596bb910258f91: Status 404 returned error can't find the container with id 3d91aeaecf0ebefe2981906f746401f6dc99e03b0af6a86a83596bb910258f91
Oct 30 23:59:44.493764 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.493723    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:44.493839 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.493821    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:44.493890 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:44.493874    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:59:45.429687 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:45.429650    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" event={"ID":"418c94f0-9f39-4846-b629-808c772500fc","Type":"ContainerStarted","Data":"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"}
Oct 30 23:59:45.431372 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:45.431336    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" event={"ID":"418c94f0-9f39-4846-b629-808c772500fc","Type":"ContainerStarted","Data":"3d91aeaecf0ebefe2981906f746401f6dc99e03b0af6a86a83596bb910258f91"}
Oct 30 23:59:45.599546 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:45.599469    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podStartSLOduration=2.599445639 podStartE2EDuration="2.599445639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:45.597661183 +0000 UTC m=+1037.944645082" watchObservedRunningTime="2026-10-30 23:59:45.599445639 +0000 UTC m=+1037.946429540"
Oct 30 23:59:46.029055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:46.028505    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:46.029055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:46.028631    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:46.029055 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:46.028675    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:46.972081 ip-10-0-139-66 kubenswrapper[6033]: E1030 23:59:46.971995    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 30 23:59:47.022680 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:47.021657    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:47.022680 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:47.021717    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:48.021090 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:48.021019    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:48.021564 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:48.021109    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:49.021736 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:49.021692    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:49.022257 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:49.021775    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:50.021666 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:50.021633    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:50.021857 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:50.021693    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:51.021539 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:51.021495    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 23:59:51.021747 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:51.021576    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 23:59:52.026593 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:52.026563    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:52.027058 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:52.027024    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:52.027734 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:52.027717    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 30 23:59:59.588236 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.588198    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 30 23:59:59.592656 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.592634    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.639664 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639625    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.639844 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639770    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.639891 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639865    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2s5\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.639953 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639939    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.640299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639964    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.640299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.639982    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.640299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.640112    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.640299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.640134    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.640299 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.640155    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.741848 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.741806    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2s5\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.741879    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.741907    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.741935    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.741991    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742018    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742094 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742065    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742394 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742203    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742394 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742325    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742895 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742772    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.742895 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742856    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.743231 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.742987    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.743483 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.743452    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.743745 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.743722    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.745388 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.745340    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.745493 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.745451    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.785177 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.785087    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 30 23:59:59.800132 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.800094    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2s5\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.800280 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.800168    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\") pod \"e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") " pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 30 23:59:59.907563 ip-10-0-139-66 kubenswrapper[6033]: I1030 23:59:59.907475    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:00:00.230520 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.230387    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:00.235985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.235958    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:00.236186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.236090    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.239972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.239949    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.247683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.247656    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:00.251932 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.251911    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.300991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.298477    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:00.302377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.302348    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:00.344253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.344210    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 31 00:00:00.348313 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:00:00.348284    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37b9ae7_6ac7_43e4_8157_84545067dd7f.slice/crio-962795fde54b91c79e5d40ede7ac2e18246928ba4e33b3fad006e4e6bcd1adc2 WatchSource:0}: Error finding container 962795fde54b91c79e5d40ede7ac2e18246928ba4e33b3fad006e4e6bcd1adc2: Status 404 returned error can't find the container with id 962795fde54b91c79e5d40ede7ac2e18246928ba4e33b3fad006e4e6bcd1adc2
Oct 31 00:00:00.350421 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350395    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnp9\" (UniqueName: \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.350563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350546    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.350753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350707    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rtg\" (UniqueName: \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.350879 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350864    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9vv\" (UniqueName: \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.350942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350929    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.351008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.350968    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.352617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.352576    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:00:00.352912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.352668    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:00:00.352912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.352711    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:00:00.361208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.361179    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:00.387693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.387670    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:00:00.452637 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452602    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9vv\" (UniqueName: \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.452828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452663    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.452828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452693    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.452828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452733    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnp9\" (UniqueName: \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.452828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452762    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.453058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.452875    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rtg\" (UniqueName: \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.453644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.453612    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.453872 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.453830    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.453981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.453965    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.511790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.511729    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnp9\" (UniqueName: \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.511954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.511859    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rtg\" (UniqueName: \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.512194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.512171    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9vv\" (UniqueName: \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.528331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.528294    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" event={"ID":"e37b9ae7-6ac7-43e4-8157-84545067dd7f","Type":"ContainerStarted","Data":"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"}
Oct 31 00:00:00.528487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.528362    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" event={"ID":"e37b9ae7-6ac7-43e4-8157-84545067dd7f","Type":"ContainerStarted","Data":"962795fde54b91c79e5d40ede7ac2e18246928ba4e33b3fad006e4e6bcd1adc2"}
Oct 31 00:00:00.579461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.579429    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:00.589590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.589564    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:00.604857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.604821    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:00.694014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.690723    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podStartSLOduration=1.690703549 podStartE2EDuration="1.690703549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:00.687909833 +0000 UTC m=+1053.034893725" watchObservedRunningTime="2026-10-31 00:00:00.690703549 +0000 UTC m=+1053.037687447"
Oct 31 00:00:00.908008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.907932    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:00:00.980584 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.980538    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:00.981013 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:00:00.980975    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4631fcb4_ec88_4157_a53b_51a29e342f12.slice/crio-e9a5d97aa75288faabfc7867cae68aeeb20f32faa4f5a1e317c7c5621155c6b3 WatchSource:0}: Error finding container e9a5d97aa75288faabfc7867cae68aeeb20f32faa4f5a1e317c7c5621155c6b3: Status 404 returned error can't find the container with id e9a5d97aa75288faabfc7867cae68aeeb20f32faa4f5a1e317c7c5621155c6b3
Oct 31 00:00:00.985826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:00.985737    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:01.021062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.021015    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:01.040870 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:00:01.040834    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed468d4_e417_46a0_971d_e26e6de4d84e.slice/crio-d0951645794af7365f1cdfe26ed6020070978ee25f9647f62a3a577fd150f5e9 WatchSource:0}: Error finding container d0951645794af7365f1cdfe26ed6020070978ee25f9647f62a3a577fd150f5e9: Status 404 returned error can't find the container with id d0951645794af7365f1cdfe26ed6020070978ee25f9647f62a3a577fd150f5e9
Oct 31 00:00:01.536381 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.536348    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerStarted","Data":"34796f2586f5f04c1dc488f5fd95e7809a01552bf8c8cb2d127871d5419cc004"}
Oct 31 00:00:01.536531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.536408    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerStarted","Data":"d0951645794af7365f1cdfe26ed6020070978ee25f9647f62a3a577fd150f5e9"}
Oct 31 00:00:01.537755 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.537729    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerStarted","Data":"e9a5d97aa75288faabfc7867cae68aeeb20f32faa4f5a1e317c7c5621155c6b3"}
Oct 31 00:00:01.539170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.539148    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerStarted","Data":"860da360480f927714abf3cfb447ebd088e385314e863cfadccdc5d8c1c80f3f"}
Oct 31 00:00:01.539259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.539186    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerStarted","Data":"52ac52395237c1ea124ecdb5acac7dd378adbe24a6d7151580e89a8f9b881d8e"}
Oct 31 00:00:01.653924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.653862    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" podStartSLOduration=1.653848765 podStartE2EDuration="1.653848765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.653080827 +0000 UTC m=+1054.000064727" watchObservedRunningTime="2026-10-31 00:00:01.653848765 +0000 UTC m=+1054.000832727"
Oct 31 00:00:01.767231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.767172    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" podStartSLOduration=1.7671576359999999 podStartE2EDuration="1.767157636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.766247158 +0000 UTC m=+1054.113231056" watchObservedRunningTime="2026-10-31 00:00:01.767157636 +0000 UTC m=+1054.114141533"
Oct 31 00:00:01.910246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.910158    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:01.910246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:01.910216    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:02.547219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.547179    6033 generic.go:356] "Generic (PLEG): container finished" podID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerID="34796f2586f5f04c1dc488f5fd95e7809a01552bf8c8cb2d127871d5419cc004" exitCode=60
Oct 31 00:00:02.547428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.547220    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerDied","Data":"34796f2586f5f04c1dc488f5fd95e7809a01552bf8c8cb2d127871d5419cc004"}
Oct 31 00:00:02.547558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.547543    6033 scope.go:122] "RemoveContainer" containerID="34796f2586f5f04c1dc488f5fd95e7809a01552bf8c8cb2d127871d5419cc004"
Oct 31 00:00:02.548975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.548954    6033 generic.go:356] "Generic (PLEG): container finished" podID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerID="d5bb93cec057571e4a98a3cc3387b97bd7b3ec95adcc9873d2e1a4b2b85b9932" exitCode=60
Oct 31 00:00:02.549084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.549025    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerDied","Data":"d5bb93cec057571e4a98a3cc3387b97bd7b3ec95adcc9873d2e1a4b2b85b9932"}
Oct 31 00:00:02.549338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.549323    6033 scope.go:122] "RemoveContainer" containerID="d5bb93cec057571e4a98a3cc3387b97bd7b3ec95adcc9873d2e1a4b2b85b9932"
Oct 31 00:00:02.550921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.550749    6033 generic.go:356] "Generic (PLEG): container finished" podID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerID="860da360480f927714abf3cfb447ebd088e385314e863cfadccdc5d8c1c80f3f" exitCode=60
Oct 31 00:00:02.550921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.550811    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerDied","Data":"860da360480f927714abf3cfb447ebd088e385314e863cfadccdc5d8c1c80f3f"}
Oct 31 00:00:02.551154 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.551072    6033 scope.go:122] "RemoveContainer" containerID="860da360480f927714abf3cfb447ebd088e385314e863cfadccdc5d8c1c80f3f"
Oct 31 00:00:02.907910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.907876    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:02.908491 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:02.908456    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:03.561264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.561224    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerStarted","Data":"ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"}
Oct 31 00:00:03.563902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.563788    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerStarted","Data":"ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"}
Oct 31 00:00:03.566276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.566237    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerStarted","Data":"377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"}
Oct 31 00:00:03.898475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.898211    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" podStartSLOduration=3.8981897549999998 podStartE2EDuration="3.898189755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:03.895461786 +0000 UTC m=+1056.242445686" watchObservedRunningTime="2026-10-31 00:00:03.898189755 +0000 UTC m=+1056.245173653"
Oct 31 00:00:03.909065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.908092    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:03.909065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:03.908155    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:04.575144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.575114    6033 generic.go:356] "Generic (PLEG): container finished" podID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerID="ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d" exitCode=60
Oct 31 00:00:04.575355 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.575154    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerDied","Data":"ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"}
Oct 31 00:00:04.575355 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.575228    6033 scope.go:122] "RemoveContainer" containerID="d5bb93cec057571e4a98a3cc3387b97bd7b3ec95adcc9873d2e1a4b2b85b9932"
Oct 31 00:00:04.575530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.575514    6033 scope.go:122] "RemoveContainer" containerID="ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"
Oct 31 00:00:04.575833 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:04.575816    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs_odh-ai-gateway-infra(4631fcb4-ec88-4157-a53b-51a29e342f12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12"
Oct 31 00:00:04.577757 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.577735    6033 generic.go:356] "Generic (PLEG): container finished" podID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerID="377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48" exitCode=60
Oct 31 00:00:04.577856 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.577792    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerDied","Data":"377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"}
Oct 31 00:00:04.578151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.578127    6033 scope.go:122] "RemoveContainer" containerID="377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"
Oct 31 00:00:04.578477 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:04.578455    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82_odh-ai-gateway-infra(b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f"
Oct 31 00:00:04.579833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.579810    6033 generic.go:356] "Generic (PLEG): container finished" podID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerID="ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499" exitCode=60
Oct 31 00:00:04.579912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.579846    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerDied","Data":"ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"}
Oct 31 00:00:04.580127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.580114    6033 scope.go:122] "RemoveContainer" containerID="ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"
Oct 31 00:00:04.580362 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:04.580345    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2_odh-ai-gateway-infra(6ed468d4-e417-46a0-971d-e26e6de4d84e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e"
Oct 31 00:00:04.617506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.617478    6033 scope.go:122] "RemoveContainer" containerID="860da360480f927714abf3cfb447ebd088e385314e863cfadccdc5d8c1c80f3f"
Oct 31 00:00:04.631728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.631706    6033 scope.go:122] "RemoveContainer" containerID="34796f2586f5f04c1dc488f5fd95e7809a01552bf8c8cb2d127871d5419cc004"
Oct 31 00:00:04.908054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.907930    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:04.908054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:04.907999    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:05.617850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:05.617665    6033 scope.go:122] "RemoveContainer" containerID="ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"
Oct 31 00:00:05.618382 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:05.618008    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2_odh-ai-gateway-infra(6ed468d4-e417-46a0-971d-e26e6de4d84e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e"
Oct 31 00:00:05.619503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:05.619485    6033 scope.go:122] "RemoveContainer" containerID="ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"
Oct 31 00:00:05.619800 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:05.619777    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs_odh-ai-gateway-infra(4631fcb4-ec88-4157-a53b-51a29e342f12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12"
Oct 31 00:00:05.621088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:05.621069    6033 scope.go:122] "RemoveContainer" containerID="377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"
Oct 31 00:00:05.621322 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:05.621302    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82_odh-ai-gateway-infra(b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f"
Oct 31 00:00:05.908600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:05.908498    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:05.908770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:05.908615    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:06.908609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:06.908567    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:00:06.909054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:06.908649    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 00:00:07.912896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:07.912859    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:00:07.913380 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:07.913294    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:00:07.913892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:07.913871    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:00:15.822806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:15.822773    6033 scope.go:122] "RemoveContainer" containerID="377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"
Oct 31 00:00:16.697832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:16.697794    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerStarted","Data":"865ae6297c99de2afab685b99d9c837033f89a1004d6704d709e63d15a72fa2c"}
Oct 31 00:00:16.822790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:16.822755    6033 scope.go:122] "RemoveContainer" containerID="ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"
Oct 31 00:00:16.971286 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:16.971201    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:00:17.707926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.707897    6033 generic.go:356] "Generic (PLEG): container finished" podID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerID="2742c72e6aab6a8e6f7785cf80b06207ef70784b58a45ad566ddf823db981539" exitCode=60
Oct 31 00:00:17.708086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.707967    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerDied","Data":"2742c72e6aab6a8e6f7785cf80b06207ef70784b58a45ad566ddf823db981539"}
Oct 31 00:00:17.708086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.708024    6033 scope.go:122] "RemoveContainer" containerID="ffa0ddf782430ec5f144c6c4a691c1b808d60c60a6e9ef803e66d7188ed14499"
Oct 31 00:00:17.708506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.708487    6033 scope.go:122] "RemoveContainer" containerID="2742c72e6aab6a8e6f7785cf80b06207ef70784b58a45ad566ddf823db981539"
Oct 31 00:00:17.708844 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:17.708824    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2_odh-ai-gateway-infra(6ed468d4-e417-46a0-971d-e26e6de4d84e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e"
Oct 31 00:00:17.709958 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.709936    6033 generic.go:356] "Generic (PLEG): container finished" podID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerID="865ae6297c99de2afab685b99d9c837033f89a1004d6704d709e63d15a72fa2c" exitCode=60
Oct 31 00:00:17.710071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.710001    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerDied","Data":"865ae6297c99de2afab685b99d9c837033f89a1004d6704d709e63d15a72fa2c"}
Oct 31 00:00:17.710464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.710448    6033 scope.go:122] "RemoveContainer" containerID="865ae6297c99de2afab685b99d9c837033f89a1004d6704d709e63d15a72fa2c"
Oct 31 00:00:17.710794 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:17.710774    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82_odh-ai-gateway-infra(b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f"
Oct 31 00:00:17.747013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.746991    6033 scope.go:122] "RemoveContainer" containerID="377d8348ac017425f648a1c6de84e2a98087ea602788a365d9f00092523d3e48"
Oct 31 00:00:17.819786 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.819535    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:17.821997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:17.821971    6033 scope.go:122] "RemoveContainer" containerID="ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"
Oct 31 00:00:18.719836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.719805    6033 generic.go:356] "Generic (PLEG): container finished" podID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerID="8b0e3f38610703b7465730892c0dc0dcfadc9b5885279f7a436ec412f80c4dbf" exitCode=60
Oct 31 00:00:18.720323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.719879    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerDied","Data":"8b0e3f38610703b7465730892c0dc0dcfadc9b5885279f7a436ec412f80c4dbf"}
Oct 31 00:00:18.720323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.719948    6033 scope.go:122] "RemoveContainer" containerID="ad51ea55754410b9ac2ce1e78bfbcfab218cbc380293faf90c63affd73f1c15d"
Oct 31 00:00:18.720462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.720397    6033 scope.go:122] "RemoveContainer" containerID="8b0e3f38610703b7465730892c0dc0dcfadc9b5885279f7a436ec412f80c4dbf"
Oct 31 00:00:18.720787 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:18.720763    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs_odh-ai-gateway-infra(4631fcb4-ec88-4157-a53b-51a29e342f12)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12"
Oct 31 00:00:18.839009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.838965    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:18.896021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:18.895988    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:19.017490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.017458    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:19.043553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.043517    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\" (UniqueName: \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\") pod \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") "
Oct 31 00:00:19.043708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.043678    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\") pod \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\" (UID: \"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f\") "
Oct 31 00:00:19.043995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.043970    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca" pod "b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" (UID: "b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:19.044148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.044135    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:19.045705 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.045683    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg" pod "b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" (UID: "b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f"). InnerVolumeSpecName "kube-api-access-v7rtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:19.145746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.145709    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\" (UniqueName: \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\") pod \"6ed468d4-e417-46a0-971d-e26e6de4d84e\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") "
Oct 31 00:00:19.145746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.145750    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\") pod \"6ed468d4-e417-46a0-971d-e26e6de4d84e\" (UID: \"6ed468d4-e417-46a0-971d-e26e6de4d84e\") "
Oct 31 00:00:19.146137 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.146124    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7rtg\" (UniqueName: \"kubernetes.io/projected/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f-kube-api-access-v7rtg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:19.146212 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.146186    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca" pod "6ed468d4-e417-46a0-971d-e26e6de4d84e" (UID: "6ed468d4-e417-46a0-971d-e26e6de4d84e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:19.148155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.148121    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9" pod "6ed468d4-e417-46a0-971d-e26e6de4d84e" (UID: "6ed468d4-e417-46a0-971d-e26e6de4d84e"). InnerVolumeSpecName "kube-api-access-frnp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:19.247122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.246988    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnp9\" (UniqueName: \"kubernetes.io/projected/6ed468d4-e417-46a0-971d-e26e6de4d84e-kube-api-access-frnp9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:19.247122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.247024    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed468d4-e417-46a0-971d-e26e6de4d84e-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:19.729827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.729780    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"
Oct 31 00:00:19.731288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.730102    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2" event={"ID":"6ed468d4-e417-46a0-971d-e26e6de4d84e","Type":"ContainerDied","Data":"d0951645794af7365f1cdfe26ed6020070978ee25f9647f62a3a577fd150f5e9"}
Oct 31 00:00:19.731288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.730184    6033 scope.go:122] "RemoveContainer" containerID="2742c72e6aab6a8e6f7785cf80b06207ef70784b58a45ad566ddf823db981539"
Oct 31 00:00:19.735261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.735236    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"
Oct 31 00:00:19.735402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.735280    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82" event={"ID":"b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f","Type":"ContainerDied","Data":"52ac52395237c1ea124ecdb5acac7dd378adbe24a6d7151580e89a8f9b881d8e"}
Oct 31 00:00:19.748430 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.743159    6033 scope.go:122] "RemoveContainer" containerID="865ae6297c99de2afab685b99d9c837033f89a1004d6704d709e63d15a72fa2c"
Oct 31 00:00:19.848181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.848144    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:19.911295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.911259    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:19.924424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:19.924385    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890080-dlw82"]
Oct 31 00:00:20.027678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.027647    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:20.038037 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.037993    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:20.042719 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.042684    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-27e82b-29890080-l2pd2"]
Oct 31 00:00:20.164052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.163263    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\" (UniqueName: \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\") pod \"4631fcb4-ec88-4157-a53b-51a29e342f12\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") "
Oct 31 00:00:20.164052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.163838    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\") pod \"4631fcb4-ec88-4157-a53b-51a29e342f12\" (UID: \"4631fcb4-ec88-4157-a53b-51a29e342f12\") "
Oct 31 00:00:20.164691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.164666    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca" pod "4631fcb4-ec88-4157-a53b-51a29e342f12" (UID: "4631fcb4-ec88-4157-a53b-51a29e342f12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:20.165169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.165146    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4631fcb4-ec88-4157-a53b-51a29e342f12-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:20.183563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.183501    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv" pod "4631fcb4-ec88-4157-a53b-51a29e342f12" (UID: "4631fcb4-ec88-4157-a53b-51a29e342f12"). InnerVolumeSpecName "kube-api-access-fk9vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:20.266434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.266400    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9vv\" (UniqueName: \"kubernetes.io/projected/4631fcb4-ec88-4157-a53b-51a29e342f12-kube-api-access-fk9vv\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:00:20.743527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.743444    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"
Oct 31 00:00:20.743977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.743440    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs" event={"ID":"4631fcb4-ec88-4157-a53b-51a29e342f12","Type":"ContainerDied","Data":"e9a5d97aa75288faabfc7867cae68aeeb20f32faa4f5a1e317c7c5621155c6b3"}
Oct 31 00:00:20.743977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.743593    6033 scope.go:122] "RemoveContainer" containerID="8b0e3f38610703b7465730892c0dc0dcfadc9b5885279f7a436ec412f80c4dbf"
Oct 31 00:00:20.900645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.900608    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:20.946746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:20.946706    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890080-lkgrs"]
Oct 31 00:00:21.830043 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:21.829986    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" path="/var/lib/kubelet/pods/4631fcb4-ec88-4157-a53b-51a29e342f12/volumes"
Oct 31 00:00:21.830634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:21.830613    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" path="/var/lib/kubelet/pods/6ed468d4-e417-46a0-971d-e26e6de4d84e/volumes"
Oct 31 00:00:21.831083 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:21.831064    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" path="/var/lib/kubelet/pods/b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f/volumes"
Oct 31 00:00:26.363600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:26.363564    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:00:46.470568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.470477    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:00:46.471219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471194    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471217    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471231    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471239    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471251    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471258    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471271    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471278    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471289    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471293    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471306    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471310    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471317    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471323    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471344    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.471428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471351    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471473    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471482    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471489    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471497    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471507    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:00:46.472054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.471515    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:00:46.475264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.475232    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.571211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.571173    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"openshift-service-ca.crt\""
Oct 31 00:00:46.571211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.571217    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"kube-root-ca.crt\""
Oct 31 00:00:46.571531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.571301    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"default-dockercfg-ns2t6\""
Oct 31 00:00:46.571531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.571423    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-986d37\"/\"distinct-2-w3-986d37-kserve-self-signed-certs\""
Oct 31 00:00:46.574734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.574709    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:00:46.614788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.614751    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.614972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.614817    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.614972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.614938    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.615077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.614978    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.615169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.615151    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88fw2\" (UniqueName: \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.615216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.615187    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717194    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717359    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88fw2\" (UniqueName: \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717397    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717459    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717529    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.717638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.717633    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.718569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.718538    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.718720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.718634    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.718821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.718799    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.721085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.720970    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.721617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.721588    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.777617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.777585    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88fw2\" (UniqueName: \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\") pod \"distinct-2-w3-986d37-kserve-64c85654c6-t7s98\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") " pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:46.788298 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:46.788256    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:47.008863 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:00:47.008764    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:00:47.177024 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:00:47.176982    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ea89ff8_bb44_4645_9b72_5fb32f0ee31a.slice/crio-0847babf33636ea5326feca870d5f9bfcd5eb8f37ea270d8184b60b40d3132a2 WatchSource:0}: Error finding container 0847babf33636ea5326feca870d5f9bfcd5eb8f37ea270d8184b60b40d3132a2: Status 404 returned error can't find the container with id 0847babf33636ea5326feca870d5f9bfcd5eb8f37ea270d8184b60b40d3132a2
Oct 31 00:00:47.177850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:47.177825    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:00:47.977562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:47.977510    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" event={"ID":"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a","Type":"ContainerStarted","Data":"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"}
Oct 31 00:00:47.978074 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:47.977582    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" event={"ID":"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a","Type":"ContainerStarted","Data":"0847babf33636ea5326feca870d5f9bfcd5eb8f37ea270d8184b60b40d3132a2"}
Oct 31 00:00:48.089454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:48.089402    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podStartSLOduration=2.089386734 podStartE2EDuration="2.089386734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:48.088583788 +0000 UTC m=+1100.435567686" watchObservedRunningTime="2026-10-31 00:00:48.089386734 +0000 UTC m=+1100.436370632"
Oct 31 00:00:48.789463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:48.789424    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:56.788802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:56.788760    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:56.802457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:56.802424    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:57.049203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:57.049125    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:00:58.778377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:00:58.778340    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:01:16.971156 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:01:16.971050    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:01:18.408343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.408301    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:01:18.409156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.409135    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:01:18.409156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.409157    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:01:18.409388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.409329    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88ca6c0-dff2-4068-a5f4-cbb1ef6f711f" containerName="cleanup"
Oct 31 00:01:18.409388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.409348    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4631fcb4-ec88-4157-a53b-51a29e342f12" containerName="cleanup"
Oct 31 00:01:18.409388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.409365    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed468d4-e417-46a0-971d-e26e6de4d84e" containerName="cleanup"
Oct 31 00:01:18.413225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.413203    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.512775 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.512729    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-27e82b\"/\"distinct-2-w2-27e82b-kserve-self-signed-certs\""
Oct 31 00:01:18.515586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.515555    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:01:18.555431 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555391    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.555617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555550    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.555666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555625    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.555708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555695    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.555779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555760    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.555820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.555802    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkgws\" (UniqueName: \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657368    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657448    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657477    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkgws\" (UniqueName: \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657630    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657705    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.657765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.657746    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.658535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.658130    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.658535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.658208    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.658535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.658290    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.660217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.660191    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.660462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.660443    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.716011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.715976    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkgws\" (UniqueName: \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\") pod \"distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") " pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:18.727502 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:18.727468    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:19.108874 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:19.108844    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:01:19.110579 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:01:19.110540    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52aaf67b_7519_484b_af54_a512508d32ed.slice/crio-a65f285f5ce098f056b7c9e365b82b38ae26f106b38f7ab7a24f067c630ac181 WatchSource:0}: Error finding container a65f285f5ce098f056b7c9e365b82b38ae26f106b38f7ab7a24f067c630ac181: Status 404 returned error can't find the container with id a65f285f5ce098f056b7c9e365b82b38ae26f106b38f7ab7a24f067c630ac181
Oct 31 00:01:19.185148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:19.185100    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" event={"ID":"52aaf67b-7519-484b-af54-a512508d32ed","Type":"ContainerStarted","Data":"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"}
Oct 31 00:01:19.185296 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:19.185159    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" event={"ID":"52aaf67b-7519-484b-af54-a512508d32ed","Type":"ContainerStarted","Data":"a65f285f5ce098f056b7c9e365b82b38ae26f106b38f7ab7a24f067c630ac181"}
Oct 31 00:01:19.298736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:19.298676    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podStartSLOduration=1.298661231 podStartE2EDuration="1.298661231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:19.297446913 +0000 UTC m=+1131.644430808" watchObservedRunningTime="2026-10-31 00:01:19.298661231 +0000 UTC m=+1131.645645128"
Oct 31 00:01:19.728014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:19.727971    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:28.728318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:28.728279    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:28.742412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:28.742379    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:29.276910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:29.276877    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:01:45.453895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.453818    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:01:45.457285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.457261    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.501002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.500959    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:01:45.504972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.504946    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.511269 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.511241    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5c3374-openshift-default-dockercfg-jbr44\""
Oct 31 00:01:45.549698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.549377    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:01:45.554475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.554438    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.554813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.554789    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.554910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.554867    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.554984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.554959    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvv9b\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.555075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.555018    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.555113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.555099    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.555175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.555159    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.555305 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.555286    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.555411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.555394    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.557613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.557582    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:01:45.656265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656224    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.656265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656268    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52lr\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656306    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656402    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656494    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.656558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656536    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656581    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.656777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656622    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656682    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.656777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656736    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656811    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.656979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.656895    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657011    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.657213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657142    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657184    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvv9b\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657222    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657238    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657285    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657371    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.657468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657234    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657471    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.657593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.657557    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.658127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.658100    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.659673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.659648    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.659867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.659850    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.715503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.715430    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.715503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.715497    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvv9b\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.758306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758267    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758312    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758371    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758425    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758450    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758516    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758570    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758705    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758755    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52lr\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.758889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758812    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.759005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758908    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.759005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.758965    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.759169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.759150    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.759354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.759333    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.760896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.760873    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.761055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.760943    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.772170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.772140    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:45.817227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.817183    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.817415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.817373    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52lr\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:45.819674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:45.819639    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:46.197812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.197779    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:01:46.199042 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:01:46.198997    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c92dfc0_58e8_4ac1_9dbf_4c93b379fd06.slice/crio-867a2cbef655f39225d641e994b9f0cf93d64ee7771341afd3c4efe41d0081f2 WatchSource:0}: Error finding container 867a2cbef655f39225d641e994b9f0cf93d64ee7771341afd3c4efe41d0081f2: Status 404 returned error can't find the container with id 867a2cbef655f39225d641e994b9f0cf93d64ee7771341afd3c4efe41d0081f2
Oct 31 00:01:46.201704 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.201664    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.201810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.201753    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.201810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.201802    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.256250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.256221    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:01:46.257477 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:01:46.257444    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dc8d0d_3428_4e39_9b48_61ab8cafb419.slice/crio-18705d2f44680c7188c8da024cdeb044591fd16521d19ebca158df024e141249 WatchSource:0}: Error finding container 18705d2f44680c7188c8da024cdeb044591fd16521d19ebca158df024e141249: Status 404 returned error can't find the container with id 18705d2f44680c7188c8da024cdeb044591fd16521d19ebca158df024e141249
Oct 31 00:01:46.264721 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.264662    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.264860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.264795    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.264860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.264836    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:01:46.384014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.382717    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" event={"ID":"46dc8d0d-3428-4e39-9b48-61ab8cafb419","Type":"ContainerStarted","Data":"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"}
Oct 31 00:01:46.384014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.383960    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" event={"ID":"46dc8d0d-3428-4e39-9b48-61ab8cafb419","Type":"ContainerStarted","Data":"18705d2f44680c7188c8da024cdeb044591fd16521d19ebca158df024e141249"}
Oct 31 00:01:46.387069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.386982    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" event={"ID":"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06","Type":"ContainerStarted","Data":"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"}
Oct 31 00:01:46.387069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.387062    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" event={"ID":"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06","Type":"ContainerStarted","Data":"867a2cbef655f39225d641e994b9f0cf93d64ee7771341afd3c4efe41d0081f2"}
Oct 31 00:01:46.508809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.508744    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podStartSLOduration=1.508723613 podStartE2EDuration="1.508723613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:46.506253108 +0000 UTC m=+1158.853237007" watchObservedRunningTime="2026-10-31 00:01:46.508723613 +0000 UTC m=+1158.855707511"
Oct 31 00:01:46.626434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.626362    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podStartSLOduration=1.6263390869999999 podStartE2EDuration="1.626339087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:01:46.62526898 +0000 UTC m=+1158.972252906" watchObservedRunningTime="2026-10-31 00:01:46.626339087 +0000 UTC m=+1158.973322986"
Oct 31 00:01:46.772424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.772316    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:46.820553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:46.820512    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:46.972853 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:01:46.972817    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:01:47.774687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:47.774650    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:47.775114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:47.774708    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:47.838180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:47.833675    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:47.838180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:47.833743    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:48.773111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:48.773021    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:48.773341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:48.773131    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:48.822055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:48.821493    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:48.822055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:48.821572    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:49.773216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:49.773177    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:49.773412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:49.773233    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:49.822055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:49.821451    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:49.822055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:49.821521    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:50.773402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:50.773367    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:50.773655 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:50.773632    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:50.821053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:50.820806    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:50.821053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:50.820867    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:51.773265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:51.773113    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:51.773265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:51.773172    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:51.826012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:51.825383    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:51.826012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:51.825443    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:52.772678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:52.772624    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 00:01:52.772905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:52.772692    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 00:01:52.821016 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:52.820966    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 00:01:52.821585 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:52.821086    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 00:01:53.865198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:53.865129    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:53.866215 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:53.866188    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:53.867194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:53.867170    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:01:54.054460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:54.053346    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:54.054653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:54.054633    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:01:54.055569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:01:54.055545    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:02:00.594904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.594862    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:02:00.599312 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.599276    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.701954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.701910    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:02:00.776005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.775962    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776155    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776214    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776356 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776248    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776433    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776518    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776579    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776603    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.776639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.776634    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmx7\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878171    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878269    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878311    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878336    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878423    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878479    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878552    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878585    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878635    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmx7\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.878725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.878693    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.879092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.879012    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.879092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.879059    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.879411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.879389    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.879551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.879529    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.880880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.880854    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.881174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.881153    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.936764 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.936724    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:00.936951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:00.936802    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmx7\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\") pod \"e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") " pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:01.213353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:01.213262    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:01.642957 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:01.642922    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:02:01.644249 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:02:01.644219    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf36b503_0a11_463a_b78c_368fabfc6d68.slice/crio-6d96479573f2d95dbafb75f259a2a56a1d5b1f6bb70232a0cf8c8784c543bae6 WatchSource:0}: Error finding container 6d96479573f2d95dbafb75f259a2a56a1d5b1f6bb70232a0cf8c8784c543bae6: Status 404 returned error can't find the container with id 6d96479573f2d95dbafb75f259a2a56a1d5b1f6bb70232a0cf8c8784c543bae6
Oct 31 00:02:01.650678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:01.650635    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:01.650819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:01.650713    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:01.650819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:01.650748    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:02.063477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:02.063437    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:02:02.551533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:02.551493    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" event={"ID":"cf36b503-0a11-463a-b78c-368fabfc6d68","Type":"ContainerStarted","Data":"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"}
Oct 31 00:02:02.551727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:02.551557    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" event={"ID":"cf36b503-0a11-463a-b78c-368fabfc6d68","Type":"ContainerStarted","Data":"6d96479573f2d95dbafb75f259a2a56a1d5b1f6bb70232a0cf8c8784c543bae6"}
Oct 31 00:02:02.677712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:02.677658    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podStartSLOduration=2.677640207 podStartE2EDuration="2.677640207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:02.674507534 +0000 UTC m=+1175.021491434" watchObservedRunningTime="2026-10-31 00:02:02.677640207 +0000 UTC m=+1175.024624104"
Oct 31 00:02:03.221398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:03.221359    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:03.221898 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:03.221871    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:03.222110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:03.222080    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:04.220518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:04.220473    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:04.221166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:04.221128    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:05.216053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:05.215455    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:05.216053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:05.215523    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:05.432914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:05.432862    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 00:02:06.216646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:06.216465    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:06.216863 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:06.216838    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:07.214577 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:07.214539    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:07.214981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:07.214598    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:08.047911 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:08.047860    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"]
Oct 31 00:02:08.096912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:08.096860    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f450a80-qqq2d"]
Oct 31 00:02:08.214399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:08.214361    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 00:02:08.214585 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:08.214435    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 00:02:09.676249 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:09.676198    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:09.676792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:09.676768    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:09.677736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:09.677709    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:02:09.830778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:09.830743    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dde4d4-0afd-4fa9-8627-f4d9efb386d7" path="/var/lib/kubelet/pods/f2dde4d4-0afd-4fa9-8627-f4d9efb386d7/volumes"
Oct 31 00:02:16.972043 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:02:16.971992    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:02:19.046871 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:19.046831    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"]
Oct 31 00:02:19.093845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:19.093804    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbbc0b2b-rzqqx"]
Oct 31 00:02:19.830197 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:19.830157    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc493130-2a56-4a89-bf75-49eb4ef11fea" path="/var/lib/kubelet/pods/fc493130-2a56-4a89-bf75-49eb4ef11fea/volumes"
Oct 31 00:02:28.256399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.256366    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"]
Oct 31 00:02:28.261014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.260992    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.310613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.310576    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:02:28.357111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.357081    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:02:28.360000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.359968    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"]
Oct 31 00:02:28.369590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.369560    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.369736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.369596    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jpw\" (UniqueName: \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.369870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.369843    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.471119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.471079    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.471119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.471125    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jpw\" (UniqueName: \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.471358 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.471259    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.471948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.471914    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.473490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.473472    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.529320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.529246    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jpw\" (UniqueName: \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\") pod \"maas-api-revoke-sub-19aee852-rn8t5\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.576140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.576100    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:28.950262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.950235    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"]
Oct 31 00:02:28.951564 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:02:28.951531    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8948ba50_ab14_4f89_a476_2a9921154fdf.slice/crio-717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd WatchSource:0}: Error finding container 717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd: Status 404 returned error can't find the container with id 717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd
Oct 31 00:02:28.977569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:28.977539    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5" event={"ID":"8948ba50-ab14-4f89-a476-2a9921154fdf","Type":"ContainerStarted","Data":"717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd"}
Oct 31 00:02:29.950889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:29.950849    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:02:29.954506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:29.954481    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:29.986420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:29.986384    6033 generic.go:356] "Generic (PLEG): container finished" podID="8948ba50-ab14-4f89-a476-2a9921154fdf" containerID="21454803f65d334e9626c046bfa7b64d9ba20b24b22da9f6913ac5b5448ca742" exitCode=0
Oct 31 00:02:29.986599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:29.986463    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5" event={"ID":"8948ba50-ab14-4f89-a476-2a9921154fdf","Type":"ContainerDied","Data":"21454803f65d334e9626c046bfa7b64d9ba20b24b22da9f6913ac5b5448ca742"}
Oct 31 00:02:30.093292 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093251    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093321    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxzb\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093455    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093584 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093500    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093632 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093595    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093667    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093737 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093716    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093822 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093810    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.093865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.093836    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.154921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.154817    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:02:30.195274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195232    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195282    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195330    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195403    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195436    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195494    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195531    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195595    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195671    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxzb\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195712    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.195805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.195725    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.196334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.196310    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.196663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.196637    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.196866 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.196849    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.198356 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.198310    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.198846 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.198821    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.254371 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.254336    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.254573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.254436    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxzb\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\") pod \"e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") " pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.280585 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.280556    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:30.712820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.712787    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:02:30.715760 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:02:30.715727    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8ba6a3_5c8b_403b_a452_982ae35ad9d5.slice/crio-186bc1e590a2f26a6ce35bb8ec6c25711fd42e659e9133773183f3997901c4dc WatchSource:0}: Error finding container 186bc1e590a2f26a6ce35bb8ec6c25711fd42e659e9133773183f3997901c4dc: Status 404 returned error can't find the container with id 186bc1e590a2f26a6ce35bb8ec6c25711fd42e659e9133773183f3997901c4dc
Oct 31 00:02:30.718621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.718583    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:30.718722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.718654    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:30.718722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.718687    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:30.834352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.833449    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:02:30.837797 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.837771    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:30.908048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:30.907831    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.009749    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.009788    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.009945    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010155    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010212    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010254    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010288    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010327    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.011192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.010352    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjv9\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.024131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.019572    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" event={"ID":"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5","Type":"ContainerStarted","Data":"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"}
Oct 31 00:02:31.024131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.019645    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" event={"ID":"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5","Type":"ContainerStarted","Data":"186bc1e590a2f26a6ce35bb8ec6c25711fd42e659e9133773183f3997901c4dc"}
Oct 31 00:02:31.114347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.113557    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.114347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.113608    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.114347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.113700    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.118889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.118853    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119006 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.118947    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.119016    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.119064    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.119184    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.119230    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjv9\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.119563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.119508    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.120433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.120405    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.121495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.121469    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.122055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.122016    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.122877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.122849    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.123643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.123594    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.139265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.139223    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.140455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.140394    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podStartSLOduration=2.140375786 podStartE2EDuration="2.140375786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:31.133654292 +0000 UTC m=+1203.480638190" watchObservedRunningTime="2026-10-31 00:02:31.140375786 +0000 UTC m=+1203.487359682"
Oct 31 00:02:31.177759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.177719    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjv9\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.178914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.178881    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\") pod \"e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") " pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.200743 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.200709    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:31.281301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.281224    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:31.282605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.282581    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:31.282715 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.282626    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:31.322420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.322387    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\") pod \"8948ba50-ab14-4f89-a476-2a9921154fdf\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") "
Oct 31 00:02:31.322629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.322524    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\") pod \"8948ba50-ab14-4f89-a476-2a9921154fdf\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") "
Oct 31 00:02:31.322714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.322694    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\" (UniqueName: \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\") pod \"8948ba50-ab14-4f89-a476-2a9921154fdf\" (UID: \"8948ba50-ab14-4f89-a476-2a9921154fdf\") "
Oct 31 00:02:31.322831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.322808    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca" pod "8948ba50-ab14-4f89-a476-2a9921154fdf" (UID: "8948ba50-ab14-4f89-a476-2a9921154fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:31.324207 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.323487    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:31.325324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.325290    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw" pod "8948ba50-ab14-4f89-a476-2a9921154fdf" (UID: "8948ba50-ab14-4f89-a476-2a9921154fdf"). InnerVolumeSpecName "kube-api-access-b4jpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:31.326240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.326211    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca" pod "8948ba50-ab14-4f89-a476-2a9921154fdf" (UID: "8948ba50-ab14-4f89-a476-2a9921154fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:31.424636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.424601    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4jpw\" (UniqueName: \"kubernetes.io/projected/8948ba50-ab14-4f89-a476-2a9921154fdf-kube-api-access-b4jpw\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:31.424636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.424633    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8948ba50-ab14-4f89-a476-2a9921154fdf-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:31.455885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.455845    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:31.919225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.919192    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:02:31.921227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.921189    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:31.921346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.921269    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:31.921346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:31.921299    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:02:32.026687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.026655    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"
Oct 31 00:02:32.027061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.026694    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5" event={"ID":"8948ba50-ab14-4f89-a476-2a9921154fdf","Type":"ContainerDied","Data":"717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd"}
Oct 31 00:02:32.027061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.026743    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717f5c76fd34832b41b3732a32a27b74871c9ab8db10362ce4bd5283ea9da3bd"
Oct 31 00:02:32.028653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.028620    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" event={"ID":"344316a1-caba-4f26-8d3a-3bcd2a4b89ee","Type":"ContainerStarted","Data":"8ed7e62b4b8fc6a056e60b45663e83bde6803c9c319d43e246cd87b6b9a3d8a2"}
Oct 31 00:02:32.285086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.283166    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:32.285086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:32.283228    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:33.041975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.041945    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" event={"ID":"344316a1-caba-4f26-8d3a-3bcd2a4b89ee","Type":"ContainerStarted","Data":"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"}
Oct 31 00:02:33.231111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.230587    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podStartSLOduration=3.230561892 podStartE2EDuration="3.230561892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:33.212549779 +0000 UTC m=+1205.559533678" watchObservedRunningTime="2026-10-31 00:02:33.230561892 +0000 UTC m=+1205.577545791"
Oct 31 00:02:33.299077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.298508    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:33.299077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.298577    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:33.456326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.456291    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:33.457803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.457778    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:33.457873 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:33.457831    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:34.283476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:34.283436    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:34.284116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:34.284081    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:34.456634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:34.456598    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:34.456827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:34.456655    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:35.304053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:35.300172    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:35.304053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:35.300238    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:35.456793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:35.456755    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:35.456965 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:35.456809    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:36.285471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:36.285432    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:36.285784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:36.285750    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:36.456454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:36.456412    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:36.456970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:36.456531    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:37.286529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:37.286487    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:02:37.286854 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:37.286819    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:02:37.457116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:37.457077    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:37.457598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:37.457136    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:38.457218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:38.457183    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:02:38.457785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:38.457380    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 00:02:38.797146 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:38.797112    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:38.797574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:38.797553    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:38.798420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:38.798402    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:02:40.602056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:40.594194    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:40.602056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:40.594290    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:40.872810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:40.872677    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:40.938088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:40.937144    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.010650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.010589    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.032182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.032126    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.086299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.086253    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.273208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.273157    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.279165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.279118    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.280582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.280261    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.415233 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.415171    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.424788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.424391    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 00:02:41.613114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.610165    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:41.613114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.610263    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.746056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.745192    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.905246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.905135    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.906298 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.906271    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:02:41.941198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.941135    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:41.943001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:41.941947    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:02:42.016190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.016135    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.016891 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.016865    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:02:42.042224 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.042173    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.042890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.042863    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:02:42.094064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.087266    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.095020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.094989    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:02:42.315064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.284848    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.315064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.285249    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.315064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.285765    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 31 00:02:42.315064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.285824    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:02:42.354087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.315615    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.433071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.403273    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:02:42.483659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.436398    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.483659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.474930    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.540257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.536404    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 31 00:02:42.587118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.544437    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:02:42.665446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.664728    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:42.665446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.664796    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.746057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.696426    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:02:42.766104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.746379    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:42.766104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.751198    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:02:42.913285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.913177    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Oct 31 00:02:42.999159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:42.999115    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.028058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.001259    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:02:43.028363 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.020200    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.071449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.056781    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.296056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.287117    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.321057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.312120    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 00:02:43.440187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.440137    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.552107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.548609    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.572065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.570003    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:02:43.625064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.619442    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:02:43.686685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.686637    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:02:43.722340 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.722305    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:02:43.722618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.722589    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.732098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.732057    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:43.752126 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.741477    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:02:43.752126 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:43.743080    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 31 00:02:44.111739 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.111699    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:02:44.183742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.183700    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.363168 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.363060    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:02:44.367683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.367633    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 00:02:44.423053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.418147    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:02:44.472116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.471160    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:02:44.475284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.475242    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 31 00:02:44.673285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.673207    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:44.673545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.673525    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:44.674409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:44.674387    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:02:46.972765 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:02:46.972724    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:02:49.657669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.657631    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"]
Oct 31 00:02:49.658319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.658303    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8948ba50-ab14-4f89-a476-2a9921154fdf" containerName="revoke-keys"
Oct 31 00:02:49.658385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.658320    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8948ba50-ab14-4f89-a476-2a9921154fdf" containerName="revoke-keys"
Oct 31 00:02:49.658463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.658453    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8948ba50-ab14-4f89-a476-2a9921154fdf" containerName="revoke-keys"
Oct 31 00:02:49.662405 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.662382    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.712612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.712567    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:02:49.758924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.758882    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:02:49.759225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.759193    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.759350 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.759271    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.759618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.759582    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vm4w\" (UniqueName: \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.762489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.762461    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"]
Oct 31 00:02:49.861391 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.861349    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vm4w\" (UniqueName: \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.861591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.861462    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.861591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.861497    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.862313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.862291    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.864055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.864011    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.919712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.919636    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vm4w\" (UniqueName: \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\") pod \"maas-api-revoke-sub-473a7013-k2qxg\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:49.976509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:49.976472    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:50.350833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:50.350800    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"]
Oct 31 00:02:50.352497 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:02:50.352462    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5e9846_3db5_4e84_af64_c89fe2b16636.slice/crio-fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba WatchSource:0}: Error finding container fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba: Status 404 returned error can't find the container with id fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba
Oct 31 00:02:50.617942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:50.617902    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg" event={"ID":"df5e9846-3db5-4e84-af64-c89fe2b16636","Type":"ContainerStarted","Data":"fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba"}
Oct 31 00:02:51.630285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:51.630249    6033 generic.go:356] "Generic (PLEG): container finished" podID="df5e9846-3db5-4e84-af64-c89fe2b16636" containerID="a05cfd74a1388da1d700e585e311d477293d4d04fa7eb506a648d0bd8a3b2bdc" exitCode=0
Oct 31 00:02:51.630705 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:51.630318    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg" event={"ID":"df5e9846-3db5-4e84-af64-c89fe2b16636","Type":"ContainerDied","Data":"a05cfd74a1388da1d700e585e311d477293d4d04fa7eb506a648d0bd8a3b2bdc"}
Oct 31 00:02:52.804780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.804740    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:52.905083 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.904953    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\") pod \"df5e9846-3db5-4e84-af64-c89fe2b16636\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") "
Oct 31 00:02:52.905083 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.905020    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\") pod \"df5e9846-3db5-4e84-af64-c89fe2b16636\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") "
Oct 31 00:02:52.905295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.905213    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\" (UniqueName: \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\") pod \"df5e9846-3db5-4e84-af64-c89fe2b16636\" (UID: \"df5e9846-3db5-4e84-af64-c89fe2b16636\") "
Oct 31 00:02:52.905525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.905492    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca" pod "df5e9846-3db5-4e84-af64-c89fe2b16636" (UID: "df5e9846-3db5-4e84-af64-c89fe2b16636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:52.906309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.906280    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:52.907306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.907281    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w" pod "df5e9846-3db5-4e84-af64-c89fe2b16636" (UID: "df5e9846-3db5-4e84-af64-c89fe2b16636"). InnerVolumeSpecName "kube-api-access-5vm4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:52.907421 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:52.907401    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca" pod "df5e9846-3db5-4e84-af64-c89fe2b16636" (UID: "df5e9846-3db5-4e84-af64-c89fe2b16636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:53.008505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:53.008462    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vm4w\" (UniqueName: \"kubernetes.io/projected/df5e9846-3db5-4e84-af64-c89fe2b16636-kube-api-access-5vm4w\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:53.008505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:53.008504    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5e9846-3db5-4e84-af64-c89fe2b16636-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:02:53.645553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:53.645510    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg" event={"ID":"df5e9846-3db5-4e84-af64-c89fe2b16636","Type":"ContainerDied","Data":"fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba"}
Oct 31 00:02:53.645553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:53.645538    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"
Oct 31 00:02:53.645774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:53.645571    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbad5d1cf05c6b9cbf57ec24407bba9cb5ce297d5b25a5349634d8dbd43a93ba"
Oct 31 00:02:59.788264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.788221    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:02:59.788976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.788961    6033 cpu_manager.go:414] "containerMap: removing container" podUID="df5e9846-3db5-4e84-af64-c89fe2b16636" containerName="revoke-keys"
Oct 31 00:02:59.789056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.788978    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5e9846-3db5-4e84-af64-c89fe2b16636" containerName="revoke-keys"
Oct 31 00:02:59.789110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.789100    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5e9846-3db5-4e84-af64-c89fe2b16636" containerName="revoke-keys"
Oct 31 00:02:59.793151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.793125    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808500    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808556    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808674    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808704    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808824    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.808897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.808848    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh499\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.809046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.809008    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.809111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.809072    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.809152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.809104    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.851210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.851173    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:02:59.910009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.909966    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910012    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910121    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910147    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910251    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910287    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh499\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910490    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910555    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910584    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910592    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.910933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.910905    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.911290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.911258    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.911290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.911275    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.911469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.911400    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.912819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.912786    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.913214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.913193    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.968357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.968323    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:02:59.968783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:02:59.968766    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh499\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:00.108353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.108258    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:00.555476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.555441    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:03:00.557485 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:00.557455    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0e7bdf_b088_47c0_8e28_3fb78307b77d.slice/crio-d9b9b94f3ea9f2c3fe6c10ea43210736e936b458acc588438e0a9bb0aa32ab0b WatchSource:0}: Error finding container d9b9b94f3ea9f2c3fe6c10ea43210736e936b458acc588438e0a9bb0aa32ab0b: Status 404 returned error can't find the container with id d9b9b94f3ea9f2c3fe6c10ea43210736e936b458acc588438e0a9bb0aa32ab0b
Oct 31 00:03:00.560145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.560103    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:00.560236 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.560179    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:00.560274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.560233    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:00.696834 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.696795    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" event={"ID":"3e0e7bdf-b088-47c0-8e28-3fb78307b77d","Type":"ContainerStarted","Data":"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"}
Oct 31 00:03:00.697094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.696858    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" event={"ID":"3e0e7bdf-b088-47c0-8e28-3fb78307b77d","Type":"ContainerStarted","Data":"d9b9b94f3ea9f2c3fe6c10ea43210736e936b458acc588438e0a9bb0aa32ab0b"}
Oct 31 00:03:00.812854 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:00.812746    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podStartSLOduration=1.812728361 podStartE2EDuration="1.812728361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:00.811485999 +0000 UTC m=+1233.158469898" watchObservedRunningTime="2026-10-31 00:03:00.812728361 +0000 UTC m=+1233.159712322"
Oct 31 00:03:01.076142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.076046    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 31 00:03:01.076357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.076319    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy" containerID="cri-o://cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8" gracePeriod=30
Oct 31 00:03:01.081783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.081757    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 31 00:03:01.081991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.081953    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy" containerID="cri-o://5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd" gracePeriod=30
Oct 31 00:03:01.108944 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.108903    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:01.110598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.110569    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:01.110746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.110629    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:01.125331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.125289    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 31 00:03:01.125695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:01.125662    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy" containerID="cri-o://5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719" gracePeriod=30
Oct 31 00:03:02.109775 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:02.109464    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:02.109775 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:02.109566    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:03.109437 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:03.109400    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:03.109749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:03.109459    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:04.111194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:04.111156    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:04.111819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:04.111775    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:05.109121 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.109081    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:05.109306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.109158    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:05.870163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.870125    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"]
Oct 31 00:03:05.876150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.876121    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:05.925705 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.925622    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:05.926019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.925987    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:03:05.929243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:05.929211    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"]
Oct 31 00:03:06.025390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.025357    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.025744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.025726    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62hv\" (UniqueName: \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.025914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.025903    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.112477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.112413    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:06.112572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.112474    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:06.128361 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.128324    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62hv\" (UniqueName: \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.128528 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.128510    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.128652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.128636    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.129963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.129931    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.131561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.131534    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.212313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.212132    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62hv\" (UniqueName: \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\") pod \"maas-api-revoke-sub-95470bd7-vtxhb\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.220977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.220582    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:06.559133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.559104    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 31 00:03:06.562568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.562542    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 31 00:03:06.632057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.632014    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 31 00:03:06.640597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640570    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.640747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640612    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.640747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640717    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.640816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640751    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.640816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640796    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.640937 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640916    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.640991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.640977    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641022    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641075    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.641122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641100    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.641122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641107    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641158    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641195    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.641283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641241    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.641283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641276    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.641486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641315    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641383    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.641486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641436    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.641486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641471    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\") pod \"f98204b8-0c6c-49a3-acc4-5d90832930e4\" (UID: \"f98204b8-0c6c-49a3-acc4-5d90832930e4\") "
Oct 31 00:03:06.641706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641540    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.641706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641588    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\") pod \"e09448aa-e522-4109-8e39-a7e295f4237f\" (UID: \"e09448aa-e522-4109-8e39-a7e295f4237f\") "
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641382    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641472    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641672    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641887    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.641944    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.642231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642200    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:06.642609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642450    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.642828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642802    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.642908 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642833    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.642908 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642848    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642862    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642935    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642951    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642966    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.642988    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643004    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f98204b8-0c6c-49a3-acc4-5d90832930e4-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643018    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e09448aa-e522-4109-8e39-a7e295f4237f-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.643607 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643576    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:03:06.643720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643605    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:03:06.643720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643616    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.643842 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.643765    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.644417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.644390    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "kube-api-access-7thtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.644504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.644466    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token" pod "f98204b8-0c6c-49a3-acc4-5d90832930e4" (UID: "f98204b8-0c6c-49a3-acc4-5d90832930e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.645147 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.645127    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.645242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.645200    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn" pod "e09448aa-e522-4109-8e39-a7e295f4237f" (UID: "e09448aa-e522-4109-8e39-a7e295f4237f"). InnerVolumeSpecName "kube-api-access-ngtxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.743690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743602    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.743859 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743684    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.743859 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743731    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.743859 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743773    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.743859 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743813    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.744107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743879    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.744107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743898    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.744107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.743980    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.744211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.744183    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.744323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.744302    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:06.744417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.744400    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.744474 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.744449    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.744627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.744612    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\") pod \"d905008c-4172-400f-bbdf-64473060620f\" (UID: \"d905008c-4172-400f-bbdf-64473060620f\") "
Oct 31 00:03:06.745096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745057    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.745537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745501    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745543    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngtxn\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-kube-api-access-ngtxn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745561    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e09448aa-e522-4109-8e39-a7e295f4237f-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745575    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745589    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745603    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f98204b8-0c6c-49a3-acc4-5d90832930e4-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745622    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745635    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.745650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745648    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d905008c-4172-400f-bbdf-64473060620f-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.746022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745663    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e09448aa-e522-4109-8e39-a7e295f4237f-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.746022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745677    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e09448aa-e522-4109-8e39-a7e295f4237f-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.746022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745693    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thtc\" (UniqueName: \"kubernetes.io/projected/f98204b8-0c6c-49a3-acc4-5d90832930e4-kube-api-access-7thtc\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.746022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.745706    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.746896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.746869    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:03:06.746896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.746877    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "kube-api-access-gchcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.748113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.748085    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:03:06.748370 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.748351    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token" pod "d905008c-4172-400f-bbdf-64473060620f" (UID: "d905008c-4172-400f-bbdf-64473060620f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:06.751529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.751503    6033 generic.go:356] "Generic (PLEG): container finished" podID="e09448aa-e522-4109-8e39-a7e295f4237f" containerID="5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719" exitCode=0
Oct 31 00:03:06.751636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.751544    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"
Oct 31 00:03:06.751636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.751581    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" event={"ID":"e09448aa-e522-4109-8e39-a7e295f4237f","Type":"ContainerDied","Data":"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"}
Oct 31 00:03:06.751747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.751718    6033 scope.go:122] "RemoveContainer" containerID="5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"
Oct 31 00:03:06.752040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.752001    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5" event={"ID":"e09448aa-e522-4109-8e39-a7e295f4237f","Type":"ContainerDied","Data":"702a7f55dc9ecb6e50abbd83fa92d88ec139363b1587e4130c368f2483465e61"}
Oct 31 00:03:06.758156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.758129    6033 generic.go:356] "Generic (PLEG): container finished" podID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerID="5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd" exitCode=0
Oct 31 00:03:06.758449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.758428    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" event={"ID":"f98204b8-0c6c-49a3-acc4-5d90832930e4","Type":"ContainerDied","Data":"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"}
Oct 31 00:03:06.758662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.758644    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd" event={"ID":"f98204b8-0c6c-49a3-acc4-5d90832930e4","Type":"ContainerDied","Data":"58feece8513115befa0d7db85feb469c2c5584a777a7e28730a672bc9a2f1546"}
Oct 31 00:03:06.758939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.758924    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"
Oct 31 00:03:06.760689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.760655    6033 generic.go:356] "Generic (PLEG): container finished" podID="d905008c-4172-400f-bbdf-64473060620f" containerID="cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8" exitCode=0
Oct 31 00:03:06.760795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.760717    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" event={"ID":"d905008c-4172-400f-bbdf-64473060620f","Type":"ContainerDied","Data":"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"}
Oct 31 00:03:06.760795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.760770    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj" event={"ID":"d905008c-4172-400f-bbdf-64473060620f","Type":"ContainerDied","Data":"81cbbb08e46db06aa6b67b974bac925f4aa859dc3762ca6a8776801e006d9d48"}
Oct 31 00:03:06.760910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.760885    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"
Oct 31 00:03:06.776676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.776653    6033 scope.go:122] "RemoveContainer" containerID="5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"
Oct 31 00:03:06.777118 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:06.777093    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719\": container with ID starting with 5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719 not found: ID does not exist" containerID="5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"
Oct 31 00:03:06.777218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.777123    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719"} err="failed to get container status \"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719\": rpc error: code = NotFound desc = could not find container \"5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719\": container with ID starting with 5c9db3321af1cc538ccb61a5dd36d56b0b5bddc09e4041bdb6ba483e3b9e1719 not found: ID does not exist"
Oct 31 00:03:06.777218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.777141    6033 scope.go:122] "RemoveContainer" containerID="5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"
Oct 31 00:03:06.800677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.791013    6033 scope.go:122] "RemoveContainer" containerID="5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"
Oct 31 00:03:06.800677 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:06.791465    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd\": container with ID starting with 5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd not found: ID does not exist" containerID="5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"
Oct 31 00:03:06.800677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.791506    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd"} err="failed to get container status \"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd\": rpc error: code = NotFound desc = could not find container \"5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd\": container with ID starting with 5bfedbf91b0a0c40301837c79d0d0ec7d2c0f4dfbc31667b2d15dfd74c5d97dd not found: ID does not exist"
Oct 31 00:03:06.800677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.791532    6033 scope.go:122] "RemoveContainer" containerID="cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"
Oct 31 00:03:06.819895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.819865    6033 scope.go:122] "RemoveContainer" containerID="cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"
Oct 31 00:03:06.820290 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:06.820247    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8\": container with ID starting with cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8 not found: ID does not exist" containerID="cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"
Oct 31 00:03:06.820404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.820290    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8"} err="failed to get container status \"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8\": rpc error: code = NotFound desc = could not find container \"cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8\": container with ID starting with cac31f375d3036f882f19b7505be453897e565cdc8c66032bc771dc9cde7ced8 not found: ID does not exist"
Oct 31 00:03:06.846875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.846838    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchcc\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-kube-api-access-gchcc\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.846875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.846870    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d905008c-4172-400f-bbdf-64473060620f-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.846875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.846879    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d905008c-4172-400f-bbdf-64473060620f-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.846875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.846888    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d905008c-4172-400f-bbdf-64473060620f-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:06.867536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.867504    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"]
Oct 31 00:03:06.868955 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:06.868928    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54701d7_b389_465d_b274_f44903692999.slice/crio-cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a WatchSource:0}: Error finding container cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a: Status 404 returned error can't find the container with id cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a
Oct 31 00:03:06.987760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.987721    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7132-264395"]
Oct 31 00:03:06.988711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988678    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy"
Oct 31 00:03:06.988711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988711    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy"
Oct 31 00:03:06.988861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988727    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy"
Oct 31 00:03:06.988861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988735    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy"
Oct 31 00:03:06.988861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988792    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy"
Oct 31 00:03:06.988861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988799    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy"
Oct 31 00:03:06.988995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988946    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" containerName="istio-proxy"
Oct 31 00:03:06.988995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988972    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d905008c-4172-400f-bbdf-64473060620f" containerName="istio-proxy"
Oct 31 00:03:06.988995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.988984    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" containerName="istio-proxy"
Oct 31 00:03:06.994914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:06.994855    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:07.016537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.016504    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 31 00:03:07.022075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.022040    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-m59d5"]
Oct 31 00:03:07.042150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.042115    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7132-264395"]
Oct 31 00:03:07.044173 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.044141    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bghhz\""
Oct 31 00:03:07.057627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.053692    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\") pod \"test-curl-7132-264395\" (UID: \"b12c12e0-6f1a-4b93-8578-f9101ece39db\") " pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:07.109579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.109548    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:03:07.109770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.109598    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 31 00:03:07.127471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.127421    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 31 00:03:07.132480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.132449    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-q6mtd"]
Oct 31 00:03:07.155573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.155528    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\") pod \"test-curl-7132-264395\" (UID: \"b12c12e0-6f1a-4b93-8578-f9101ece39db\") " pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:07.213772 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.213732    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\") pod \"test-curl-7132-264395\" (UID: \"b12c12e0-6f1a-4b93-8578-f9101ece39db\") " pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:07.235462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.234937    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 31 00:03:07.239352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.239324    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-cxzjj"]
Oct 31 00:03:07.308872 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.308827    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:07.686166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.685860    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7132-264395"]
Oct 31 00:03:07.686166 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:07.686065    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb12c12e0_6f1a_4b93_8578_f9101ece39db.slice/crio-969b43d83f9dce8541f00aa18af007f1f6c390b5f16df8c4a2c20b6704d29e01 WatchSource:0}: Error finding container 969b43d83f9dce8541f00aa18af007f1f6c390b5f16df8c4a2c20b6704d29e01: Status 404 returned error can't find the container with id 969b43d83f9dce8541f00aa18af007f1f6c390b5f16df8c4a2c20b6704d29e01
Oct 31 00:03:07.688706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.688684    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:03:07.773754 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.773505    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb" event={"ID":"b54701d7-b389-465d-b274-f44903692999","Type":"ContainerStarted","Data":"8431d3eeb886fa7618f25a9a7bb21531702f4aa7535383ed23cb407b2d984fd5"}
Oct 31 00:03:07.773754 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.773568    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb" event={"ID":"b54701d7-b389-465d-b274-f44903692999","Type":"ContainerStarted","Data":"cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a"}
Oct 31 00:03:07.775938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.775912    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7132-264395" event={"ID":"b12c12e0-6f1a-4b93-8578-f9101ece39db","Type":"ContainerStarted","Data":"969b43d83f9dce8541f00aa18af007f1f6c390b5f16df8c4a2c20b6704d29e01"}
Oct 31 00:03:07.831809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.831776    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d905008c-4172-400f-bbdf-64473060620f" path="/var/lib/kubelet/pods/d905008c-4172-400f-bbdf-64473060620f/volumes"
Oct 31 00:03:07.832268 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.832254    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09448aa-e522-4109-8e39-a7e295f4237f" path="/var/lib/kubelet/pods/e09448aa-e522-4109-8e39-a7e295f4237f/volumes"
Oct 31 00:03:07.832679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.832666    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98204b8-0c6c-49a3-acc4-5d90832930e4" path="/var/lib/kubelet/pods/f98204b8-0c6c-49a3-acc4-5d90832930e4/volumes"
Oct 31 00:03:07.882973 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:07.882920    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb" podStartSLOduration=2.882905336 podStartE2EDuration="2.882905336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:07.881908354 +0000 UTC m=+1240.228892265" watchObservedRunningTime="2026-10-31 00:03:07.882905336 +0000 UTC m=+1240.229889228"
Oct 31 00:03:08.132844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:08.131891    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:08.132844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:08.132352    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:08.133553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:08.133485    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:03:08.784892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:08.784862    6033 generic.go:356] "Generic (PLEG): container finished" podID="b54701d7-b389-465d-b274-f44903692999" containerID="8431d3eeb886fa7618f25a9a7bb21531702f4aa7535383ed23cb407b2d984fd5" exitCode=0
Oct 31 00:03:08.785271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:08.784954    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb" event={"ID":"b54701d7-b389-465d-b274-f44903692999","Type":"ContainerDied","Data":"8431d3eeb886fa7618f25a9a7bb21531702f4aa7535383ed23cb407b2d984fd5"}
Oct 31 00:03:09.793591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:09.793550    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7132-264395" event={"ID":"b12c12e0-6f1a-4b93-8578-f9101ece39db","Type":"ContainerStarted","Data":"fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796"}
Oct 31 00:03:09.902260 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:09.902199    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7132-264395" podStartSLOduration=2.6957126430000002 podStartE2EDuration="3.902178083s" totalImagesPullingTime="1.20646544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:09.900884902 +0000 UTC m=+1242.247868805" watchObservedRunningTime="2026-10-31 00:03:09.902178083 +0000 UTC m=+1242.249161982"
Oct 31 00:03:09.976246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:09.976219    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:10.111219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.111185    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\") pod \"b54701d7-b389-465d-b274-f44903692999\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") "
Oct 31 00:03:10.111409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.111289    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\") pod \"b54701d7-b389-465d-b274-f44903692999\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") "
Oct 31 00:03:10.111409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.111397    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\" (UniqueName: \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\") pod \"b54701d7-b389-465d-b274-f44903692999\" (UID: \"b54701d7-b389-465d-b274-f44903692999\") "
Oct 31 00:03:10.111749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.111710    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca" pod "b54701d7-b389-465d-b274-f44903692999" (UID: "b54701d7-b389-465d-b274-f44903692999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:10.112041 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.112006    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54701d7-b389-465d-b274-f44903692999-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:10.116088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.116054    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca" pod "b54701d7-b389-465d-b274-f44903692999" (UID: "b54701d7-b389-465d-b274-f44903692999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:10.116282 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.116252    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv" pod "b54701d7-b389-465d-b274-f44903692999" (UID: "b54701d7-b389-465d-b274-f44903692999"). InnerVolumeSpecName "kube-api-access-c62hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:10.213870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.213838    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54701d7-b389-465d-b274-f44903692999-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:10.213870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.213868    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c62hv\" (UniqueName: \"kubernetes.io/projected/b54701d7-b389-465d-b274-f44903692999-kube-api-access-c62hv\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:10.801827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.801792    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"
Oct 31 00:03:10.802284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.801828    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb" event={"ID":"b54701d7-b389-465d-b274-f44903692999","Type":"ContainerDied","Data":"cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a"}
Oct 31 00:03:10.802284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:10.801888    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb8d3f2ca2285ba5c0bf3c46fe56675b9570e3eb6aa6c2bd768c30616e874f2a"
Oct 31 00:03:11.336954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.336914    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7132-264395"]
Oct 31 00:03:11.345022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.344993    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7132-264395"]
Oct 31 00:03:11.552930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.552897    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7132-26d505"]
Oct 31 00:03:11.553550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553532    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b54701d7-b389-465d-b274-f44903692999" containerName="revoke-keys"
Oct 31 00:03:11.553630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553551    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54701d7-b389-465d-b274-f44903692999" containerName="revoke-keys"
Oct 31 00:03:11.553630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553585    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" containerName="test-curl-7132-264395"
Oct 31 00:03:11.553630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553590    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" containerName="test-curl-7132-264395"
Oct 31 00:03:11.553788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553754    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54701d7-b389-465d-b274-f44903692999" containerName="revoke-keys"
Oct 31 00:03:11.553788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.553778    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" containerName="test-curl-7132-264395"
Oct 31 00:03:11.557713 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.557695    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:11.626925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.626843    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7132-26d505"]
Oct 31 00:03:11.636965 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.636931    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlgb\" (UniqueName: \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\") pod \"test-curl-7132-26d505\" (UID: \"99cec407-0daf-4092-ad91-b198172cef5c\") " pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:11.738832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.738791    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlgb\" (UniqueName: \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\") pod \"test-curl-7132-26d505\" (UID: \"99cec407-0daf-4092-ad91-b198172cef5c\") " pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:11.796878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.796834    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlgb\" (UniqueName: \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\") pod \"test-curl-7132-26d505\" (UID: \"99cec407-0daf-4092-ad91-b198172cef5c\") " pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:11.810933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.810901    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7132-264395" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" containerName="test-curl-7132-264395" containerID="cri-o://fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796" gracePeriod=2
Oct 31 00:03:11.859069 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:11.858994    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-264395\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:11.870666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:11.870629    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:12.245853 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:12.245817    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7132-26d505"]
Oct 31 00:03:12.247279 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:12.247221    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99cec407_0daf_4092_ad91_b198172cef5c.slice/crio-2fc2bbaef7447f380f5ff7759f3819f308dc889043cbb6af77b24bc9a66d99cb WatchSource:0}: Error finding container 2fc2bbaef7447f380f5ff7759f3819f308dc889043cbb6af77b24bc9a66d99cb: Status 404 returned error can't find the container with id 2fc2bbaef7447f380f5ff7759f3819f308dc889043cbb6af77b24bc9a66d99cb
Oct 31 00:03:12.819998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:12.819951    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7132-26d505" event={"ID":"99cec407-0daf-4092-ad91-b198172cef5c","Type":"ContainerStarted","Data":"3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df"}
Oct 31 00:03:12.820403 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:12.820009    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7132-26d505" event={"ID":"99cec407-0daf-4092-ad91-b198172cef5c","Type":"ContainerStarted","Data":"2fc2bbaef7447f380f5ff7759f3819f308dc889043cbb6af77b24bc9a66d99cb"}
Oct 31 00:03:12.927498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:12.927445    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7132-26d505" podStartSLOduration=1.927432367 podStartE2EDuration="1.927432367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:12.926310105 +0000 UTC m=+1245.273294003" watchObservedRunningTime="2026-10-31 00:03:12.927432367 +0000 UTC m=+1245.274416265"
Oct 31 00:03:14.080395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.080324    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:14.173501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.173469    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\") pod \"b12c12e0-6f1a-4b93-8578-f9101ece39db\" (UID: \"b12c12e0-6f1a-4b93-8578-f9101ece39db\") "
Oct 31 00:03:14.175641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.175609    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd" pod "b12c12e0-6f1a-4b93-8578-f9101ece39db" (UID: "b12c12e0-6f1a-4b93-8578-f9101ece39db"). InnerVolumeSpecName "kube-api-access-xd7cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:14.275445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.275408    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7cd\" (UniqueName: \"kubernetes.io/projected/b12c12e0-6f1a-4b93-8578-f9101ece39db-kube-api-access-xd7cd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:14.424066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.423737    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7132-26d505"]
Oct 31 00:03:14.427625 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.427594    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7132-26d505"]
Oct 31 00:03:14.836258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.836223    6033 generic.go:356] "Generic (PLEG): container finished" podID="b12c12e0-6f1a-4b93-8578-f9101ece39db" containerID="fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796" exitCode=137
Oct 31 00:03:14.836463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.836274    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:14.836463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.836327    6033 scope.go:122] "RemoveContainer" containerID="fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796"
Oct 31 00:03:14.836574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.836531    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7132-26d505" podUID="99cec407-0daf-4092-ad91-b198172cef5c" containerName="test-curl-7132-26d505" containerID="cri-o://3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df" gracePeriod=2
Oct 31 00:03:14.847765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.847742    6033 scope.go:122] "RemoveContainer" containerID="fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796"
Oct 31 00:03:14.848111 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:14.848088    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796\": container with ID starting with fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796 not found: ID does not exist" containerID="fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796"
Oct 31 00:03:14.848174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:14.848127    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796"} err="failed to get container status \"fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796\": rpc error: code = NotFound desc = could not find container \"fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796\": container with ID starting with fdf332cad9339e483bbafb27c67131b7d9bd9cf7139ed583acc9c517ba762796 not found: ID does not exist"
Oct 31 00:03:14.884656 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:14.884617    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-26d505\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="99cec407-0daf-4092-ad91-b198172cef5c" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:14.932510 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:14.932466    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-26d505\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="99cec407-0daf-4092-ad91-b198172cef5c" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:14.980289 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:14.980248    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-264395\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:15.087163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.087076    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"]
Oct 31 00:03:15.087772 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.087752    6033 cpu_manager.go:414] "containerMap: removing container" podUID="99cec407-0daf-4092-ad91-b198172cef5c" containerName="test-curl-7132-26d505"
Oct 31 00:03:15.087865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.087773    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cec407-0daf-4092-ad91-b198172cef5c" containerName="test-curl-7132-26d505"
Oct 31 00:03:15.087952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.087940    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cec407-0daf-4092-ad91-b198172cef5c" containerName="test-curl-7132-26d505"
Oct 31 00:03:15.092784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.092751    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.141096 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:15.141021    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-264395\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" pod="opendatahub/test-curl-7132-264395"
Oct 31 00:03:15.142318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.142292    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:15.143887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.143862    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"]
Oct 31 00:03:15.186411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.186378    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:03:15.186612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.186421    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.186612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.186462    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9dc\" (UniqueName: \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.186766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.186744    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.188950 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:15.188917    6033 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7132-26d505\" is forbidden: User \"system:node:ip-10-0-139-66.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-66.us-west-2.compute.internal' and this object" podUID="99cec407-0daf-4092-ad91-b198172cef5c" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:15.288133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.288095    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.288307 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.288207    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.288307 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.288234    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9dc\" (UniqueName: \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.288788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.288765    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.290734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.290708    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.346675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.346601    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9dc\" (UniqueName: \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\") pod \"maas-api-revoke-sub-75c3c974-m958w\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.407670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.407633    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:15.790817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.790784    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"]
Oct 31 00:03:15.792090 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:15.792020    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac4db9f_7c88_43e9_89f3_4b843bf7050a.slice/crio-1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6 WatchSource:0}: Error finding container 1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6: Status 404 returned error can't find the container with id 1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6
Oct 31 00:03:15.834709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.834645    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12c12e0-6f1a-4b93-8578-f9101ece39db" path="/var/lib/kubelet/pods/b12c12e0-6f1a-4b93-8578-f9101ece39db/volumes"
Oct 31 00:03:15.848067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:15.847990    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w" event={"ID":"9ac4db9f-7c88-43e9-89f3-4b843bf7050a","Type":"ContainerStarted","Data":"1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6"}
Oct 31 00:03:16.857736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:16.857707    6033 generic.go:356] "Generic (PLEG): container finished" podID="9ac4db9f-7c88-43e9-89f3-4b843bf7050a" containerID="fb5cb826b7415eae18d6304da2789f4826002bfab63334506da3abac38e7812b" exitCode=0
Oct 31 00:03:16.858087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:16.857771    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w" event={"ID":"9ac4db9f-7c88-43e9-89f3-4b843bf7050a","Type":"ContainerDied","Data":"fb5cb826b7415eae18d6304da2789f4826002bfab63334506da3abac38e7812b"}
Oct 31 00:03:16.971445 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:16.971233    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:03:17.103544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.103511    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:17.127267 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.127232    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\" (UniqueName: \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\") pod \"99cec407-0daf-4092-ad91-b198172cef5c\" (UID: \"99cec407-0daf-4092-ad91-b198172cef5c\") "
Oct 31 00:03:17.129394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.129366    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb" pod "99cec407-0daf-4092-ad91-b198172cef5c" (UID: "99cec407-0daf-4092-ad91-b198172cef5c"). InnerVolumeSpecName "kube-api-access-hnlgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:17.229639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.229547    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlgb\" (UniqueName: \"kubernetes.io/projected/99cec407-0daf-4092-ad91-b198172cef5c-kube-api-access-hnlgb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:17.647484 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.647449    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:03:17.831610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.831575    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cec407-0daf-4092-ad91-b198172cef5c" path="/var/lib/kubelet/pods/99cec407-0daf-4092-ad91-b198172cef5c/volumes"
Oct 31 00:03:17.865689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.865653    6033 generic.go:356] "Generic (PLEG): container finished" podID="99cec407-0daf-4092-ad91-b198172cef5c" containerID="3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df" exitCode=137
Oct 31 00:03:17.866134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.865697    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7132-26d505"
Oct 31 00:03:17.866134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.865748    6033 scope.go:122] "RemoveContainer" containerID="3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df"
Oct 31 00:03:17.876553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.876530    6033 scope.go:122] "RemoveContainer" containerID="3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df"
Oct 31 00:03:17.876908 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:17.876882    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df\": container with ID starting with 3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df not found: ID does not exist" containerID="3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df"
Oct 31 00:03:17.876978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:17.876923    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df"} err="failed to get container status \"3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df\": rpc error: code = NotFound desc = could not find container \"3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df\": container with ID starting with 3a90e069974dcf22620778c88ee8fdad8255ac5e08b1191d5435c1a7c1fe98df not found: ID does not exist"
Oct 31 00:03:18.032424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.032397    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:18.144020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.143977    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\") pod \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") "
Oct 31 00:03:18.144245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.144059    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\" (UniqueName: \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\") pod \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") "
Oct 31 00:03:18.144320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.144277    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\") pod \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\" (UID: \"9ac4db9f-7c88-43e9-89f3-4b843bf7050a\") "
Oct 31 00:03:18.152351 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.152296    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca" pod "9ac4db9f-7c88-43e9-89f3-4b843bf7050a" (UID: "9ac4db9f-7c88-43e9-89f3-4b843bf7050a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:18.168353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.168287    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc" pod "9ac4db9f-7c88-43e9-89f3-4b843bf7050a" (UID: "9ac4db9f-7c88-43e9-89f3-4b843bf7050a"). InnerVolumeSpecName "kube-api-access-jq9dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:18.180761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.178257    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca" pod "9ac4db9f-7c88-43e9-89f3-4b843bf7050a" (UID: "9ac4db9f-7c88-43e9-89f3-4b843bf7050a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:18.246697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.246658    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:18.246697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.246688    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:18.246697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.246699    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9dc\" (UniqueName: \"kubernetes.io/projected/9ac4db9f-7c88-43e9-89f3-4b843bf7050a-kube-api-access-jq9dc\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:03:18.876314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.876280    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w" event={"ID":"9ac4db9f-7c88-43e9-89f3-4b843bf7050a","Type":"ContainerDied","Data":"1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6"}
Oct 31 00:03:18.876314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.876299    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"
Oct 31 00:03:18.876896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:18.876331    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd59d76a6cb6f84cb5ca8caf0e5ccd8f39d099d04c82626a3820b1e41aab7a6"
Oct 31 00:03:21.565055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:21.564469    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.617056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:21.614143    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:21.746224 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:21.746168    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.048252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.048200    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.322266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.322161    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:03:23.322570 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.322532    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.418903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.418790    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.485204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.485153    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.606286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.606178    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.667208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.654159    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 00:03:23.681394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.654258    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:23.772307 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.772206    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:03:23.772642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:23.772586    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.107055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.066403    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.107055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.068095    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:03:24.116220 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.116163    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Oct 31 00:03:24.425138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.425010    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.461066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.434590    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:03:24.534242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.534168    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.653379 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.630409    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:03:24.698274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.676050    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.699582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.697409    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.700069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.700013    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:03:24.700212 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.700133    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:03:24.700726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.700697    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.868218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.868186    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:03:24.905323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.905288    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:03:24.952204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.952069    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:24.992045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:24.992002    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:03:25.882899 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.882854    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:03:25.883835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.883806    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9ac4db9f-7c88-43e9-89f3-4b843bf7050a" containerName="revoke-keys"
Oct 31 00:03:25.883835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.883835    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac4db9f-7c88-43e9-89f3-4b843bf7050a" containerName="revoke-keys"
Oct 31 00:03:25.884066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.884052    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac4db9f-7c88-43e9-89f3-4b843bf7050a" containerName="revoke-keys"
Oct 31 00:03:25.888676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.888645    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:25.951275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.951229    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:03:25.985979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:25.985932    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8e7d0098-openshift-default-dockercfg-5jg95\""
Oct 31 00:03:26.021882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.021845    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.021904    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022179 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022112    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9sw4\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022292 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022272    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022404    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022465    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022579    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022659    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.022749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.022718    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124326    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9sw4\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124402    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124470    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124514    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124624    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124683    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124709    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.124800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124759    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124811    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124945    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.124971    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.125215    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.125279    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.125343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.125316    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.127187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.127163    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.127579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.127553    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.183479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.183402    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.183611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.183598    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9sw4\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\") pod \"e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") " pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.213072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.210444    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:26.666025 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.665987    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:03:26.700045 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:26.700002    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ef6ff_fae1_4271_b157_3345c0fe108c.slice/crio-4a25e963dbdec9992967e3fef53f01c15e5e29b1c84967ffb325c85642517928 WatchSource:0}: Error finding container 4a25e963dbdec9992967e3fef53f01c15e5e29b1c84967ffb325c85642517928: Status 404 returned error can't find the container with id 4a25e963dbdec9992967e3fef53f01c15e5e29b1c84967ffb325c85642517928
Oct 31 00:03:26.703802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.703748    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:26.703909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.703858    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:26.703981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:26.703909    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:27.456784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:27.456744    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerStarted","Data":"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"}
Oct 31 00:03:27.457195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:27.456806    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerStarted","Data":"4a25e963dbdec9992967e3fef53f01c15e5e29b1c84967ffb325c85642517928"}
Oct 31 00:03:27.624151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:27.624086    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podStartSLOduration=2.6240727980000003 podStartE2EDuration="2.624072798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:27.621980872 +0000 UTC m=+1259.968964771" watchObservedRunningTime="2026-10-31 00:03:27.624072798 +0000 UTC m=+1259.971056771"
Oct 31 00:03:28.038014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.037979    6033 scope.go:122] "RemoveContainer" containerID="d98936319ed0d4157b2f7bcaf333a6c532da8d916b026da21516a2ac4440b349"
Oct 31 00:03:28.049017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.048990    6033 scope.go:122] "RemoveContainer" containerID="450343b38b9dba1dfffc86afc7ff4d5ea21c3c3ee8d6ce0d72b7f629e5f440f9"
Oct 31 00:03:28.212135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.212094    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:28.218329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.218293    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:28.463493 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.463402    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:28.464533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:28.464510    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:03:39.491128 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.491011    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:03:39.497019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.496995    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.567996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.567952    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e0751b65-openshift-default-dockercfg-nv4xp\""
Oct 31 00:03:39.620317 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.618235    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677627    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrz5\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677707    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677808    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677878    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677908    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.677979    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.678243    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.678328    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.678507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.678369    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780234    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780343    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780427    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrz5\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780473    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780516    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780584    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780611    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780638    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780808    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.780898    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.781264    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.781432    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.781533    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.781995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.781715    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.784433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.784405    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.785024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.785001    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.841014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.840975    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:39.841299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:39.841261    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrz5\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\") pod \"e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") " pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:40.111433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.111316    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:40.541122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.538074    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:03:40.541122 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:40.540494    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b06a5d7_9314_4cc0_863a_01c1d10c0ebb.slice/crio-83fe5427a6ed7c4009e1a046f2ae2ecd765546218f5ad24f92337dadd1ebd797 WatchSource:0}: Error finding container 83fe5427a6ed7c4009e1a046f2ae2ecd765546218f5ad24f92337dadd1ebd797: Status 404 returned error can't find the container with id 83fe5427a6ed7c4009e1a046f2ae2ecd765546218f5ad24f92337dadd1ebd797
Oct 31 00:03:40.544108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.544064    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:40.544250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.544150    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:40.544250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.544190    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:40.606248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:40.606207    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerStarted","Data":"83fe5427a6ed7c4009e1a046f2ae2ecd765546218f5ad24f92337dadd1ebd797"}
Oct 31 00:03:41.614795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:41.614749    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerStarted","Data":"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"}
Oct 31 00:03:41.823489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:41.823434    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podStartSLOduration=2.82341809 podStartE2EDuration="2.82341809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:41.775325208 +0000 UTC m=+1274.122309157" watchObservedRunningTime="2026-10-31 00:03:41.82341809 +0000 UTC m=+1274.170401997"
Oct 31 00:03:42.112402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:42.112369    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:42.118416 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:42.118384    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:42.622104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:42.622068    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:42.623233 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:42.623205    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:03:44.897016 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.896974    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:03:44.901884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.901858    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.943811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.943775    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:03:44.948075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.948052    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:44.965610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965574    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.965776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965636    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.965776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965716    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.965868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965804    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.965905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965869    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.965992 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.965975    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.966072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.966059    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.966131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.966118    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx57s\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.966209 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.966192    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:44.990834 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:44.990799    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:03:45.016825 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.016789    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:03:45.067610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067573    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.067807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067622    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.067807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067646    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.067807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067695    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.067807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067748    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.067807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067796    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067833    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067900    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.068094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067941    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx57s\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.067975    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068103    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068155    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068184    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068242    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tx4q\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.068510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068298    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.068510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068348    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.068510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068402    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.068510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.068444    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.069110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.069016    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.069334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.069305    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.069534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.069494    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.069652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.069631    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.070000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.069970    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.071477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.071451    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.071564 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.071516    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.161829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.161754    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx57s\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.164072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.164011    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.170600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170568    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tx4q\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170651    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170699    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170741    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170769    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170793    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170842    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.170914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170898    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.171012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.170995    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.171545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.171520    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.172245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.172223    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.172839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.172760    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.172905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.172885    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.173171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.173143    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.174379 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.174360    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.174697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.174680    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.219086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.219051    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:03:45.219309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.219284    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:45.224801 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.224493    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.239045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.239004    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.239165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.239132    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tx4q\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\") pod \"e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") " pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.263065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.263007    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:45.276978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.276944    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277162 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277017    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277162 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277138    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277216    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277352    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277446    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277477    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6rg\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277580    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.277675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.277662    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.296635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.296600    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:03:45.380005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.379969    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380146    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380238    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380271    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380330    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380356    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6rg\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380417    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380522 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380502    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380618    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.380951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.380925    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.381076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.381053    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.381194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.381167    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.381280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.381253    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.381835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.381812    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.383546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.383522    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.383744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.383710    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.442375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.442289    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.445246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.445210    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6rg\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\") pod \"e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") " pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.562861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.562827    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:45.669082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.668529    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:03:45.701546 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:45.701502    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69081c7_fca0_4523_b36b_2d412f6e2807.slice/crio-eb8ee5ff55144e7c30532c5b045d621ae2ee6307f5d4bed512769c2c2b37dc8f WatchSource:0}: Error finding container eb8ee5ff55144e7c30532c5b045d621ae2ee6307f5d4bed512769c2c2b37dc8f: Status 404 returned error can't find the container with id eb8ee5ff55144e7c30532c5b045d621ae2ee6307f5d4bed512769c2c2b37dc8f
Oct 31 00:03:45.704926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.704882    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:45.705088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.704958    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:45.705088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.704995    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:45.718411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.718381    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:03:45.723516 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:45.723472    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb1b044_c66c_42f8_884f_665fa0639fab.slice/crio-654a3b521cd9792204c3bc442b263446de3fe724de19bed6c0a66bbeb00723f4 WatchSource:0}: Error finding container 654a3b521cd9792204c3bc442b263446de3fe724de19bed6c0a66bbeb00723f4: Status 404 returned error can't find the container with id 654a3b521cd9792204c3bc442b263446de3fe724de19bed6c0a66bbeb00723f4
Oct 31 00:03:45.733372 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.733323    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:45.733495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.733434    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:45.733495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:45.733480    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:46.035132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.017803    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:03:46.038620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.038149    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:46.038620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.038250    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:46.038620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.038301    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:46.665894 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.665847    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" event={"ID":"9fb1b044-c66c-42f8-884f-665fa0639fab","Type":"ContainerStarted","Data":"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"}
Oct 31 00:03:46.666208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.665967    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" event={"ID":"9fb1b044-c66c-42f8-884f-665fa0639fab","Type":"ContainerStarted","Data":"654a3b521cd9792204c3bc442b263446de3fe724de19bed6c0a66bbeb00723f4"}
Oct 31 00:03:46.668259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.668226    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" event={"ID":"e69081c7-fca0-4523-b36b-2d412f6e2807","Type":"ContainerStarted","Data":"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"}
Oct 31 00:03:46.668403 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.668283    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" event={"ID":"e69081c7-fca0-4523-b36b-2d412f6e2807","Type":"ContainerStarted","Data":"eb8ee5ff55144e7c30532c5b045d621ae2ee6307f5d4bed512769c2c2b37dc8f"}
Oct 31 00:03:46.670277 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.670245    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" event={"ID":"6966b8f0-0cbc-45d7-ae91-73f210fc3a39","Type":"ContainerStarted","Data":"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"}
Oct 31 00:03:46.670392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.670295    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" event={"ID":"6966b8f0-0cbc-45d7-ae91-73f210fc3a39","Type":"ContainerStarted","Data":"d4bf2fc915948deab33fb83b943adb042bc24828a63329b572e67362e84788c9"}
Oct 31 00:03:46.803056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.784864    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podStartSLOduration=2.784840959 podStartE2EDuration="2.784840959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:46.782221793 +0000 UTC m=+1279.129205695" watchObservedRunningTime="2026-10-31 00:03:46.784840959 +0000 UTC m=+1279.131824858"
Oct 31 00:03:46.903268 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:46.903198    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podStartSLOduration=2.903175271 podStartE2EDuration="2.903175271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:46.901937916 +0000 UTC m=+1279.248921815" watchObservedRunningTime="2026-10-31 00:03:46.903175271 +0000 UTC m=+1279.250159171"
Oct 31 00:03:47.049009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.048936    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podStartSLOduration=2.048909811 podStartE2EDuration="2.048909811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:47.0196229 +0000 UTC m=+1279.366606835" watchObservedRunningTime="2026-10-31 00:03:47.048909811 +0000 UTC m=+1279.395893710"
Oct 31 00:03:47.077825 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:03:47.077797    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:03:47.227093 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.221443    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:47.227444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.227411    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:47.227621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.227593    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:47.264190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.263807    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:47.265428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.265402    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:47.265527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.265455    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:47.564140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.563653    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:47.567662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.567625    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:47.567839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:47.567692    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:48.285924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.285888    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:48.297494 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.286537    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:48.322096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.303936    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:48.322096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.304000    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:48.501054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.498280    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:03:48.563113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.556710    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.579226 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.579169    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:48.586097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.579492    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:48.649529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.637983    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e0751b65-dockercfg-rxqnn\""
Oct 31 00:03:48.649807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.645491    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e0751b65\""
Oct 31 00:03:48.694053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.692508    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:03:48.764058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.745390    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.764058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.745733    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqwb\" (UniqueName: \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.874991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.874892    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:48.878699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.878666    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.880017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.879991    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.880798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.880771    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqwb\" (UniqueName: \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:48.906482 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.906134    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:48.929793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.929748    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:48.961873 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.961828    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:48.993056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:48.982743    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqwb\" (UniqueName: \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\") pod \"payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") " pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:49.017051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.012459    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:49.120174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.118846    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:49.242344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.233103    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:49.242344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.233173    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:49.281429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.275112    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:49.281429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.275177    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:49.470200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.458123    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:49.596077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.593753    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:49.596077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.593813    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:49.662083 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.650369    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:49.722148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.713656    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:03:49.970513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:49.969818    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:03:50.149097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.115302    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:03:50.149097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.115381    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:03:50.162675 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:50.162428    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4290c22_faab_475c_8464_574daff9caf9.slice/crio-37dc191708813cea9cc74b17e8ac010a8994c210d4383744cd954b1acaed9f08 WatchSource:0}: Error finding container 37dc191708813cea9cc74b17e8ac010a8994c210d4383744cd954b1acaed9f08: Status 404 returned error can't find the container with id 37dc191708813cea9cc74b17e8ac010a8994c210d4383744cd954b1acaed9f08
Oct 31 00:03:50.164682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.164634    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:03:50.325054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.318178    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:50.325054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.318246    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:50.325054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.318748    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:50.325054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.318798    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:50.577665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.576641    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:50.577665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:50.576718    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:51.178840 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.178804    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" event={"ID":"b4290c22-faab-475c-8464-574daff9caf9","Type":"ContainerStarted","Data":"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"}
Oct 31 00:03:51.178992 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.178857    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" event={"ID":"b4290c22-faab-475c-8464-574daff9caf9","Type":"ContainerStarted","Data":"37dc191708813cea9cc74b17e8ac010a8994c210d4383744cd954b1acaed9f08"}
Oct 31 00:03:51.179832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.179799    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:03:51.266640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.266602    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:51.270176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.270137    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:51.270955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.270913    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:51.271112 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.270972    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:51.567059 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.564407    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:51.567059 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:51.564473    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:52.240058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.235664    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:52.240058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.235749    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:52.270185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.270144    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:52.270378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.270213    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:52.563850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.563811    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:52.564073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:52.563876    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:53.220835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.220792    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 00:03:53.221025 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.220864    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 00:03:53.263801 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.263760    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 00:03:53.264244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.263833    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 00:03:53.565982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.565941    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 00:03:53.566354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:53.566320    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 00:03:54.253966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.253934    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:54.254425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.254401    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:54.255733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.255710    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:03:54.269492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.269459    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:54.269929 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.269802    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:54.270809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.270780    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:03:54.421966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.421910    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" podStartSLOduration=6.421893128 podStartE2EDuration="6.421893128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:51.331806381 +0000 UTC m=+1283.678790280" watchObservedRunningTime="2026-10-31 00:03:54.421893128 +0000 UTC m=+1286.768877094"
Oct 31 00:03:54.568923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.568847    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:54.569208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.569184    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:54.570084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:54.570024    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:03:56.190533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.190491    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:03:56.195701 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.195668    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253238    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253385    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253461    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253541    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253574    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253616    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.253777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.253647    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.254496 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.254424    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.254974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.254700    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.254974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.254771    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:03:56.293377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.293331    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d32a7cc0-gw-openshift-default-dockercfg-sb78r\""
Oct 31 00:03:56.356175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356141    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356400 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356262    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356400 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356335    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356404    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356487    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356525    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356595    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356636    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356663    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356714    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.356889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356783    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.357281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.356998    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.357399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.357369    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.357590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.357570    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.358913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.358889    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.359326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.359296    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.415403 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.415365    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.415603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.415477    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\") pod \"e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") " pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.512731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.512687    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:56.948950 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.948916    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:03:56.951758 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:03:56.951722    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7445965_9cd7_489c_aa9d_14b624f119cb.slice/crio-cf08abee823077a0f8e6fc3f0eda7d82116ceba5205cd2688b3b332d45864dea WatchSource:0}: Error finding container cf08abee823077a0f8e6fc3f0eda7d82116ceba5205cd2688b3b332d45864dea: Status 404 returned error can't find the container with id cf08abee823077a0f8e6fc3f0eda7d82116ceba5205cd2688b3b332d45864dea
Oct 31 00:03:56.959790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.959722    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:56.960166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.960136    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:56.960600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:56.960540    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:03:57.327020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:57.326976    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerStarted","Data":"d1d830b22c87b0692ebf6fc2704d00e2932853d491682550c946887de3e74f8b"}
Oct 31 00:03:57.327448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:57.327114    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerStarted","Data":"cf08abee823077a0f8e6fc3f0eda7d82116ceba5205cd2688b3b332d45864dea"}
Oct 31 00:03:57.444481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:57.444420    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podStartSLOduration=1.444405092 podStartE2EDuration="1.444405092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:57.443409222 +0000 UTC m=+1289.790393116" watchObservedRunningTime="2026-10-31 00:03:57.444405092 +0000 UTC m=+1289.791388990"
Oct 31 00:03:57.513040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:57.512986    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:58.519572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:58.519532    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:59.342472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:59.342436    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:03:59.343680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:03:59.343652    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:04:03.345288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.345249    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:04:03.590056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.589592    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.635362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.601975    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded"
Oct 31 00:04:03.656056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.652142    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:03.902118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.889536    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.012056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:03.995169    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.012056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.000044    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.012056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.000087    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.025340 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.025267    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 00:04:04.041086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.026259    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.104058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.103141    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.466055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.465271    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.512431 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.498186    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.577180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.565131    6033 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Oct 31 00:04:04.595211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.593317    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 00:04:04.595211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.593845    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:04:04.675407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.675360    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:04.676123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.676098    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:04:04.718471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:04.717987    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 31 00:04:05.023389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.002469    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.023389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.004822    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.052057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.040943    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:04:05.084252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.084221    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:04:05.084498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.084482    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:04:05.084673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.084657    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:04:05.085753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.052347    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.087019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.068014    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.088132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.069093    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:04:05.096102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.088582    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:04:05.101201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.101173    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:04:05.117392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.116246    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:04:05.117392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.117208    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:04:05.117392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.117247    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:04:05.127061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.118680    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:04:05.194054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.191667    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:04:05.303901 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:05.303810    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:04:10.962413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:10.962375    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:04:10.968084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:10.968055    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.018533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.018495    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8e7d0098-serving-cert\""
Oct 31 00:04:11.022947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.022911    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:04:11.059549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.059496    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.059549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.059544    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.059836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.059679    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.059899 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.059858    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84xt\" (UniqueName: \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.161322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.161276    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84xt\" (UniqueName: \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.161540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.161450    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.161540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.161489    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.161667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.161581    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.161721 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:11.161657    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-serving-cert: secret "maas-api-e2e-ait-8e7d0098-serving-cert" not found
Oct 31 00:04:11.161763 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:11.161735    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls podName:b7223597-3978-48e0-9914-d1b19cb1615b nodeName:}" failed. No retries permitted until 2026-10-31 00:04:11.661713621 +0000 UTC m=+1304.008697515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls") pod "maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" (UID: "b7223597-3978-48e0-9914-d1b19cb1615b") : secret "maas-api-e2e-ait-8e7d0098-serving-cert" not found
Oct 31 00:04:11.162167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.162143    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.164377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.164350    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.257807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.257762    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84xt\" (UniqueName: \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.671493 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.671394    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.673923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.673899    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") pod \"maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:11.884677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:11.884641    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:12.267309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:12.266516    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:04:12.270304 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:04:12.270274    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7223597_3978_48e0_9914_d1b19cb1615b.slice/crio-38d847db32f1d061246687cc321449b91a7c216bb74115b3d0e421d45aea6b2b WatchSource:0}: Error finding container 38d847db32f1d061246687cc321449b91a7c216bb74115b3d0e421d45aea6b2b: Status 404 returned error can't find the container with id 38d847db32f1d061246687cc321449b91a7c216bb74115b3d0e421d45aea6b2b
Oct 31 00:04:13.141770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:13.141733    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" event={"ID":"b7223597-3978-48e0-9914-d1b19cb1615b","Type":"ContainerStarted","Data":"38d847db32f1d061246687cc321449b91a7c216bb74115b3d0e421d45aea6b2b"}
Oct 31 00:04:14.153746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:14.153696    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" event={"ID":"b7223597-3978-48e0-9914-d1b19cb1615b","Type":"ContainerStarted","Data":"da7c7d69b8e67b38bfb7b2507c199e99c6280dba03f24b176d204afc84207274"}
Oct 31 00:04:14.154338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:14.153814    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:14.267480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:14.267424    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" podStartSLOduration=3.338114064 podStartE2EDuration="4.267409526s" totalImagesPullingTime="929.295462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:14.266261734 +0000 UTC m=+1306.613245645" watchObservedRunningTime="2026-10-31 00:04:14.267409526 +0000 UTC m=+1306.614393430"
Oct 31 00:04:15.739848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.739816    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:04:15.744551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.744524    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.839733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.839682    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.839949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.839841    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.840048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.839951    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.840121 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.840023    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266jt\" (UniqueName: \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.843921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.843893    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0d96587a-serving-cert\""
Oct 31 00:04:15.847951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.847914    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:04:15.942227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.942188    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.942463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.942295    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.942463 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:15.942362    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-serving-cert: secret "maas-api-e2e-ait-0d96587a-serving-cert" not found
Oct 31 00:04:15.942463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.942400    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.942463 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:15.942445    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls podName:daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324 nodeName:}" failed. No retries permitted until 2026-10-31 00:04:16.442421554 +0000 UTC m=+1308.789405461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls") pod "maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" (UID: "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324") : secret "maas-api-e2e-ait-0d96587a-serving-cert" not found
Oct 31 00:04:15.942682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.942568    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266jt\" (UniqueName: \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.943142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.943119    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:15.945188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:15.945162    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:16.007599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.007119    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266jt\" (UniqueName: \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:16.451961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.451857    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:16.454645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.454610    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") pod \"maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:16.659652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.659611    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:16.694257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.694217    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:04:16.700390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.700359    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.753648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.753617    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:04:16.794806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.794771    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b8qjx\""
Oct 31 00:04:16.858772 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.858733    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8sqg\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.858935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.858848    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.858935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.858875    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.858935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.858907    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.859113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.859088    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.859163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.859149    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.859197 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.859175    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.859357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.859336    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.859399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.859378    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961201    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961432 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961292    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961432 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961411    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961555 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961481    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961555 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961526    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961681    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961735    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.961809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961793    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sqg\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.961903    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962241 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.962147    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962241 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.962188    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.962241    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.962333    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.962582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.962477    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.964382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.964287    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.964506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:16.964458    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:16.973869 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:16.973835    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:04:17.019426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.019385    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:17.019582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.019481    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sqg\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\") pod \"maas-default-gateway-openshift-default-8f6f87f59-thj8z\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:17.021363 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.021330    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:17.043729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.043694    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:04:17.047234 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:04:17.047197    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa300c3_3d7a_4e3c_8fd0_fd8ec0a6d324.slice/crio-033b56be58f244c12e243a1fd6a5d7f8953c7e71d9fc3df6a8a0fa9397846bd7 WatchSource:0}: Error finding container 033b56be58f244c12e243a1fd6a5d7f8953c7e71d9fc3df6a8a0fa9397846bd7: Status 404 returned error can't find the container with id 033b56be58f244c12e243a1fd6a5d7f8953c7e71d9fc3df6a8a0fa9397846bd7
Oct 31 00:04:17.182837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.182796    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" event={"ID":"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324","Type":"ContainerStarted","Data":"033b56be58f244c12e243a1fd6a5d7f8953c7e71d9fc3df6a8a0fa9397846bd7"}
Oct 31 00:04:17.411964 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.411930    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:04:17.412653 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:04:17.412629    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02390209_e090_495c_841a_78944e9e8f69.slice/crio-ab8ac3d6e158ca374042ee26642cdd9d55c3b0ab0858c5fe635316ef6a87bd01 WatchSource:0}: Error finding container ab8ac3d6e158ca374042ee26642cdd9d55c3b0ab0858c5fe635316ef6a87bd01: Status 404 returned error can't find the container with id ab8ac3d6e158ca374042ee26642cdd9d55c3b0ab0858c5fe635316ef6a87bd01
Oct 31 00:04:17.415574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.415533    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:17.415667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.415608    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:17.415667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:17.415642    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:18.191370 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.191257    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" event={"ID":"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324","Type":"ContainerStarted","Data":"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"}
Oct 31 00:04:18.191851 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.191526    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:18.193278 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.193245    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" event={"ID":"02390209-e090-495c-841a-78944e9e8f69","Type":"ContainerStarted","Data":"f3bd69855c0170b6cb7083b9ce90fb2dbcdd0c56a25152eb3923b057f5907273"}
Oct 31 00:04:18.193439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.193295    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" event={"ID":"02390209-e090-495c-841a-78944e9e8f69","Type":"ContainerStarted","Data":"ab8ac3d6e158ca374042ee26642cdd9d55c3b0ab0858c5fe635316ef6a87bd01"}
Oct 31 00:04:18.313104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.313007    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" podStartSLOduration=2.437549216 podStartE2EDuration="3.31298428s" totalImagesPullingTime="875.435064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:18.303467283 +0000 UTC m=+1310.650451182" watchObservedRunningTime="2026-10-31 00:04:18.31298428 +0000 UTC m=+1310.659968179"
Oct 31 00:04:18.419018 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:18.418961    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podStartSLOduration=2.418942713 podStartE2EDuration="2.418942713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:18.417153561 +0000 UTC m=+1310.764137462" watchObservedRunningTime="2026-10-31 00:04:18.418942713 +0000 UTC m=+1310.765926610"
Oct 31 00:04:19.022390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:19.022349    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:19.025499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:19.025457    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:19.025689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:19.025543    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:20.022176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:20.022133    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:20.022590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:20.022194    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:20.169314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:20.169280    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:04:21.021903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:21.021870    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:21.022151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:21.021923    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:22.022619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:22.022578    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:22.023158 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:22.022644    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:23.022154 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:23.022112    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:23.022320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:23.022172    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:24.022469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:24.022425    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:24.022879 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:24.022501    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:24.221663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:24.221630    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:04:25.022461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:25.022414    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:04:25.022659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:25.022477    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 00:04:26.596578 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:26.596543    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:26.596999 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:26.596960    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:26.597976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:26.597954    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:04:31.818428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.818386    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:04:31.824947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.824917    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.857366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.857329    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"]
Oct 31 00:04:31.862056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.862016    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.915457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.915418    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:04:31.972368 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972320    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45bx\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-kube-api-access-n45bx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.972567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972385    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.972567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972444    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.972696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972580    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.972696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972611    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.972696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972640    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.972696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972686    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.972953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972785    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.973094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972954    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.973094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.972984    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.973213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973099    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.973213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973140    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.973321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973212    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlhr\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.973321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973265    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.973424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973325    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.973424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973348    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.973548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973420    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:31.973548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.973462    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:31.977115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:31.977082    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"]
Oct 31 00:04:32.074521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074432    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.074793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074765    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.074936 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074809    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.074936 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074874    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.074936 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074894    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.074936 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074912    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.074980    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlhr\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075083    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075142    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075167    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075203    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075238    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075306    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45bx\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-kube-api-access-n45bx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.075338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075337    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075399    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075486    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075514    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075543    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075593    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.076000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.075653    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.076506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.076273    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.076598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.076534    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.077151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.077126    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.077297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.077204    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.078141 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.078117    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.078570 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.078548    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.079054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.078749    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.079719 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.079692    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.080222 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.080195    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.080320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.080255    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.081288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.081169    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.082933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.082903    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.134381 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.134341    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45bx\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-kube-api-access-n45bx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.136638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.136606    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba21fe6-a5e5-49c8-990b-7b3626e8b78f-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-4trk8\" (UID: \"aba21fe6-a5e5-49c8-990b-7b3626e8b78f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.136823 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.136659    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.137691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.137667    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlhr\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\") pod \"maas-default-gateway-openshift-default-8f6f87f59-wv5pf\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.142557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.142529    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:32.177630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.177595    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:32.584178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.581845    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"]
Oct 31 00:04:32.585956 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:04:32.585918    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3d636a_4076_4198_94ac_ef9bb0db2be1.slice/crio-07b857750e477a0021b967800fd17fc9df949dad524f88812defc04418c8f857 WatchSource:0}: Error finding container 07b857750e477a0021b967800fd17fc9df949dad524f88812defc04418c8f857: Status 404 returned error can't find the container with id 07b857750e477a0021b967800fd17fc9df949dad524f88812defc04418c8f857
Oct 31 00:04:32.586178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.586148    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:04:32.587434 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:04:32.587409    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba21fe6_a5e5_49c8_990b_7b3626e8b78f.slice/crio-4756b14c5003cb8fdfa757b288739b5cefec89d8354c06340d550138f67e128c WatchSource:0}: Error finding container 4756b14c5003cb8fdfa757b288739b5cefec89d8354c06340d550138f67e128c: Status 404 returned error can't find the container with id 4756b14c5003cb8fdfa757b288739b5cefec89d8354c06340d550138f67e128c
Oct 31 00:04:32.589191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.589156    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:32.589273 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.589227    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:32.589273 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.589260    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:32.590346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.590313    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:32.590441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.590375    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:32.590441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:32.590410    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:04:33.339232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.339178    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" event={"ID":"aba21fe6-a5e5-49c8-990b-7b3626e8b78f","Type":"ContainerStarted","Data":"89fc59ad03529eab54529750f50e0366e7f917f8d5662bc31607e8d3557e0827"}
Oct 31 00:04:33.339660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.339258    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" event={"ID":"aba21fe6-a5e5-49c8-990b-7b3626e8b78f","Type":"ContainerStarted","Data":"4756b14c5003cb8fdfa757b288739b5cefec89d8354c06340d550138f67e128c"}
Oct 31 00:04:33.341237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.341206    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" event={"ID":"1b3d636a-4076-4198-94ac-ef9bb0db2be1","Type":"ContainerStarted","Data":"c2853076354df537ee8817925a5589100e2958cb28a440bc3acd0a12a95f77a1"}
Oct 31 00:04:33.341364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.341266    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" event={"ID":"1b3d636a-4076-4198-94ac-ef9bb0db2be1","Type":"ContainerStarted","Data":"07b857750e477a0021b967800fd17fc9df949dad524f88812defc04418c8f857"}
Oct 31 00:04:33.470664 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.470594    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podStartSLOduration=2.470572275 podStartE2EDuration="2.470572275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:33.468658372 +0000 UTC m=+1325.815642269" watchObservedRunningTime="2026-10-31 00:04:33.470572275 +0000 UTC m=+1325.817556174"
Oct 31 00:04:33.630107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:33.629924    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podStartSLOduration=2.629901882 podStartE2EDuration="2.629901882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:33.608014321 +0000 UTC m=+1325.954998211" watchObservedRunningTime="2026-10-31 00:04:33.629901882 +0000 UTC m=+1325.976885780"
Oct 31 00:04:34.144116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.143325    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:34.145056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.144841    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:34.145056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.144912    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:34.178295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.178264    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:34.179456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.179437    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:34.179586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:34.179567    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:35.143187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:35.143148    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:35.143712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:35.143334    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:35.178601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:35.178563    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:35.178792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:35.178641    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:36.143492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:36.143444    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:36.144012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:36.143504    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:36.179049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:36.178988    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:36.179241 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:36.179068    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:37.143069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:37.143009    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:37.143366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:37.143327    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:37.179021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:37.178984    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:37.179465 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:37.179077    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:38.143700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:38.143661    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:38.143883 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:38.143720    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:38.179068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:38.179016    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:38.179566 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:38.179540    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:39.143461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:39.143417    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:39.143671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:39.143484    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:39.178694 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:39.178644    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:39.178856 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:39.178735    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:40.143592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:40.143551    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 00:04:40.144183 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:40.143626    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 00:04:40.178920 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:40.178876    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-4trk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 00:04:40.179166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:40.178947    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8" podUID="aba21fe6-a5e5-49c8-990b-7b3626e8b78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 00:04:41.651503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.651464    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:41.652334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.652304    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:41.653432 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.653406    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:04:41.708838 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.708799    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:41.709192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.709157    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:41.710533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:41.710511    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-4trk8"
Oct 31 00:04:46.976987 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:04:46.976944    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:04:51.608600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:04:51.608562    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:05:04.794604 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.794566    6033 generic.go:356] "Generic (PLEG): container finished" podID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerID="8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae" exitCode=0
Oct 31 00:05:04.795166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.794635    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerDied","Data":"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"}
Oct 31 00:05:04.795221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.795183    6033 scope.go:122] "RemoveContainer" containerID="8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"
Oct 31 00:05:04.796617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.796596    6033 generic.go:356] "Generic (PLEG): container finished" podID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerID="0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96" exitCode=0
Oct 31 00:05:04.796734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.796642    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerDied","Data":"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"}
Oct 31 00:05:04.796981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.796967    6033 scope.go:122] "RemoveContainer" containerID="0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"
Oct 31 00:05:04.797084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.797008    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:04.797159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.797133    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:04.797217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.797191    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:04.798499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.798473    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:04.798567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.798536    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:04.798607 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:04.798572    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:05.825053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:05.819179    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerStarted","Data":"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"}
Oct 31 00:05:05.837495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:05.837455    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerStarted","Data":"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"}
Oct 31 00:05:06.125386 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.122487    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:05:06.125386 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.122625    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:06.125386 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.122667    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:06.222627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.222517    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:05:06.223435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.223401    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:06.223707 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:06.223674    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:07.112324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:07.111800    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:07.112324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:07.111869    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:07.212956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:07.212909    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:07.213340 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:07.213262    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:08.112301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:08.112260    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:08.112591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:08.112560    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:08.214453 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:08.214417    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:08.214760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:08.214721    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:09.112336 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:09.112293    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:09.112635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:09.112602    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:09.226619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:09.226581    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:09.226974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:09.226942    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:10.125829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:10.125789    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:10.126988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:10.126948    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:10.226467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:10.226424    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:10.226756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:10.226724    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:11.120059 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:11.119758    6033 patch_prober.go:29] interesting pod/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 00:05:11.120059 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:11.119824    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 00:05:11.214054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:11.212459    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 00:05:11.214054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:11.212524    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 00:05:12.117150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.117120    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:05:12.117626 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.117603    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:05:12.118446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.118421    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:05:12.216680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.216647    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:05:12.217174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.216961    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:05:12.217903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:12.217884    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:05:16.971337 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:05:16.971298    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:05:30.921468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:30.921428    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:05:35.213885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:35.213126    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:05:43.242345 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.242310    6033 generic.go:356] "Generic (PLEG): container finished" podID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerID="d1d830b22c87b0692ebf6fc2704d00e2932853d491682550c946887de3e74f8b" exitCode=0
Oct 31 00:05:43.242783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.242384    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerDied","Data":"d1d830b22c87b0692ebf6fc2704d00e2932853d491682550c946887de3e74f8b"}
Oct 31 00:05:43.242882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.242866    6033 scope.go:122] "RemoveContainer" containerID="d1d830b22c87b0692ebf6fc2704d00e2932853d491682550c946887de3e74f8b"
Oct 31 00:05:43.244693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.244662    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:43.244803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.244739    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:43.244803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:43.244773    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:05:44.251753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:44.251714    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerStarted","Data":"b1747b29f1f17f325323513f7d110bfd18dcff39c6e2451cb5c0b581c064712b"}
Oct 31 00:05:44.342954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:44.342925    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:05:44.513657 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:44.513569    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:05:44.514944 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:44.514914    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:44.515102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:44.514965    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:45.513096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:45.513062    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:45.513598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:45.513119    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:46.513866 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:46.513826    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:46.514323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:46.513891    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:46.971253 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:05:46.971158    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:05:47.513668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:47.513625    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:47.513880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:47.513683    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:48.513661 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:48.513633    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:48.514169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:48.514141    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:49.513910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:49.513878    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:05:49.514355 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:49.513932    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 00:05:50.517785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:50.517749    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:05:50.518855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:05:50.518830    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:06:07.295598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.295557    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:06:07.296249 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.295775    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" podUID="b7223597-3978-48e0-9914-d1b19cb1615b" containerName="maas-api" containerID="cri-o://da7c7d69b8e67b38bfb7b2507c199e99c6280dba03f24b176d204afc84207274" gracePeriod=30
Oct 31 00:06:07.484620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.484440    6033 generic.go:356] "Generic (PLEG): container finished" podID="b7223597-3978-48e0-9914-d1b19cb1615b" containerID="da7c7d69b8e67b38bfb7b2507c199e99c6280dba03f24b176d204afc84207274" exitCode=0
Oct 31 00:06:07.484620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.484538    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" event={"ID":"b7223597-3978-48e0-9914-d1b19cb1615b","Type":"ContainerDied","Data":"da7c7d69b8e67b38bfb7b2507c199e99c6280dba03f24b176d204afc84207274"}
Oct 31 00:06:07.646836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.646806    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:06:07.665326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.665277    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\") pod \"b7223597-3978-48e0-9914-d1b19cb1615b\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") "
Oct 31 00:06:07.665520 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.665371    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\" (UniqueName: \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\") pod \"b7223597-3978-48e0-9914-d1b19cb1615b\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") "
Oct 31 00:06:07.665520 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.665454    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") pod \"b7223597-3978-48e0-9914-d1b19cb1615b\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") "
Oct 31 00:06:07.665721 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.665702    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\") pod \"b7223597-3978-48e0-9914-d1b19cb1615b\" (UID: \"b7223597-3978-48e0-9914-d1b19cb1615b\") "
Oct 31 00:06:07.666816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.666764    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca" pod "b7223597-3978-48e0-9914-d1b19cb1615b" (UID: "b7223597-3978-48e0-9914-d1b19cb1615b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:07.669511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.669464    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt" pod "b7223597-3978-48e0-9914-d1b19cb1615b" (UID: "b7223597-3978-48e0-9914-d1b19cb1615b"). InnerVolumeSpecName "kube-api-access-n84xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:07.669646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.669521    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs" pod "b7223597-3978-48e0-9914-d1b19cb1615b" (UID: "b7223597-3978-48e0-9914-d1b19cb1615b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:07.672631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.672598    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls" pod "b7223597-3978-48e0-9914-d1b19cb1615b" (UID: "b7223597-3978-48e0-9914-d1b19cb1615b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:07.770023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.769986    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7223597-3978-48e0-9914-d1b19cb1615b-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:07.770299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.770277    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:07.770421 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.770406    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84xt\" (UniqueName: \"kubernetes.io/projected/b7223597-3978-48e0-9914-d1b19cb1615b-kube-api-access-n84xt\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:07.770933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:07.770507    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7223597-3978-48e0-9914-d1b19cb1615b-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:08.417427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.417377    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:06:08.417932 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.417657    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" podUID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" containerName="maas-api" containerID="cri-o://2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068" gracePeriod=30
Oct 31 00:06:08.501819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.500478    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p" event={"ID":"b7223597-3978-48e0-9914-d1b19cb1615b","Type":"ContainerDied","Data":"38d847db32f1d061246687cc321449b91a7c216bb74115b3d0e421d45aea6b2b"}
Oct 31 00:06:08.501819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.500555    6033 scope.go:122] "RemoveContainer" containerID="da7c7d69b8e67b38bfb7b2507c199e99c6280dba03f24b176d204afc84207274"
Oct 31 00:06:08.501819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.500730    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"
Oct 31 00:06:08.668113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.665088    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:06:08.668113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.666435    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8e7d0098-d79f9f5f6-hcd5p"]
Oct 31 00:06:08.939989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:08.939927    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:06:09.048210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.043090    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\" (UniqueName: \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\") pod \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") "
Oct 31 00:06:09.048210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.043165    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") pod \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") "
Oct 31 00:06:09.048210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.043275    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\") pod \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") "
Oct 31 00:06:09.048210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.043455    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\") pod \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\" (UID: \"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324\") "
Oct 31 00:06:09.048210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.046636    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca" pod "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" (UID: "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:09.050101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.049930    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs" pod "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" (UID: "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:09.050101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.050068    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls" pod "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" (UID: "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:09.050284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.050193    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt" pod "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" (UID: "daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324"). InnerVolumeSpecName "kube-api-access-266jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:09.146323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.146284    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266jt\" (UniqueName: \"kubernetes.io/projected/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-kube-api-access-266jt\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:09.146323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.146318    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:09.146323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.146328    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:09.146637 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.146337    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:09.511797 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.511759    6033 generic.go:356] "Generic (PLEG): container finished" podID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" containerID="2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068" exitCode=0
Oct 31 00:06:09.512353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.511810    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"
Oct 31 00:06:09.512353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.511837    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" event={"ID":"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324","Type":"ContainerDied","Data":"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"}
Oct 31 00:06:09.512353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.511937    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4" event={"ID":"daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324","Type":"ContainerDied","Data":"033b56be58f244c12e243a1fd6a5d7f8953c7e71d9fc3df6a8a0fa9397846bd7"}
Oct 31 00:06:09.512353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.512005    6033 scope.go:122] "RemoveContainer" containerID="2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"
Oct 31 00:06:09.524730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.524707    6033 scope.go:122] "RemoveContainer" containerID="2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"
Oct 31 00:06:09.525135 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:06:09.525110    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068\": container with ID starting with 2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068 not found: ID does not exist" containerID="2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"
Oct 31 00:06:09.525229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.525144    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068"} err="failed to get container status \"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068\": rpc error: code = NotFound desc = could not find container \"2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068\": container with ID starting with 2bc0982ba373b243f71b25146ad85677bcf35015e409ea376c4087cdf0679068 not found: ID does not exist"
Oct 31 00:06:09.635308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.631156    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:06:09.635308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.631431    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" containerID="cri-o://015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4" gracePeriod=30
Oct 31 00:06:09.735122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.735012    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:06:09.740428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.740392    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0d96587a-5868dcdd4c-bkvw4"]
Oct 31 00:06:09.832161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.832075    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7223597-3978-48e0-9914-d1b19cb1615b" path="/var/lib/kubelet/pods/b7223597-3978-48e0-9914-d1b19cb1615b/volumes"
Oct 31 00:06:09.832715 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:09.832693    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" path="/var/lib/kubelet/pods/daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324/volumes"
Oct 31 00:06:12.224143 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:12.224101    6033 patch_prober.go:29] interesting pod/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:06:12.224634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:12.224192    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:06:14.970697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:14.970658    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:06:15.069569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069533    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069587    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069626    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069741    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069929 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069776    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069929 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069833    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.069929 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.069898    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.070098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070008    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:15.070164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070144    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.070243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070227    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\") pod \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\" (UID: \"fc4ef6ff-fae1-4271-b157-3345c0fe108c\") "
Oct 31 00:06:15.070680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070515    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:15.070680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070643    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:15.070976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.070952    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:15.071240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.071220    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.071326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.071243    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.071326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.071258    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.071326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.071272    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.071827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.071459    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:15.072733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.072695    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "kube-api-access-s9sw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:15.075943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.073248    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:15.075943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.073326    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:06:15.075943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.073950    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token" pod "fc4ef6ff-fae1-4271-b157-3345c0fe108c" (UID: "fc4ef6ff-fae1-4271-b157-3345c0fe108c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:15.176454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.176319    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9sw4\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-kube-api-access-s9sw4\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.176454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.176360    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.176454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.176377    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.176454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.176392    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.176454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.176405    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4ef6ff-fae1-4271-b157-3345c0fe108c-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:15.659245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.659201    6033 generic.go:356] "Generic (PLEG): container finished" podID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerID="015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4" exitCode=0
Oct 31 00:06:15.659428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.659237    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerDied","Data":"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"}
Oct 31 00:06:15.659428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.659269    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"
Oct 31 00:06:15.659428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.659299    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6" event={"ID":"fc4ef6ff-fae1-4271-b157-3345c0fe108c","Type":"ContainerDied","Data":"4a25e963dbdec9992967e3fef53f01c15e5e29b1c84967ffb325c85642517928"}
Oct 31 00:06:15.659428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.659325    6033 scope.go:122] "RemoveContainer" containerID="015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"
Oct 31 00:06:15.675472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.675445    6033 scope.go:122] "RemoveContainer" containerID="8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"
Oct 31 00:06:15.686915 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.686885    6033 scope.go:122] "RemoveContainer" containerID="015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"
Oct 31 00:06:15.687301 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:06:15.687277    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4\": container with ID starting with 015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4 not found: ID does not exist" containerID="015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"
Oct 31 00:06:15.687378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.687312    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4"} err="failed to get container status \"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4\": rpc error: code = NotFound desc = could not find container \"015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4\": container with ID starting with 015ab5731ced706d32d181dacf1b4eff906551702ba9d01f840b21a1a6212ca4 not found: ID does not exist"
Oct 31 00:06:15.687378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.687332    6033 scope.go:122] "RemoveContainer" containerID="8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"
Oct 31 00:06:15.687591 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:06:15.687575    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae\": container with ID starting with 8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae not found: ID does not exist" containerID="8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"
Oct 31 00:06:15.687640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.687595    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae"} err="failed to get container status \"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae\": rpc error: code = NotFound desc = could not find container \"8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae\": container with ID starting with 8e4fa00bea01da440835cdfaa8c962af854a58f772965b1fe782f8559918feae not found: ID does not exist"
Oct 31 00:06:15.889569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.889475    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:06:15.904063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:15.904008    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8e7d0098-openshift-default-7c65b994b7-ntss6"]
Oct 31 00:06:16.971292 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:06:16.971254    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:06:17.831750 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:17.831711    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" path="/var/lib/kubelet/pods/fc4ef6ff-fae1-4271-b157-3345c0fe108c/volumes"
Oct 31 00:06:34.635542 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:34.635488    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:06:45.814321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:45.814239    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:06:45.814886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:45.814489    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" containerID="cri-o://b1747b29f1f17f325323513f7d110bfd18dcff39c6e2451cb5c0b581c064712b" gracePeriod=30
Oct 31 00:06:46.090662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.090531    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:06:46.091925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.091892    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.092113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092098    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.092220 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092210    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.092288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092279    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.092373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092364    6033 cpu_manager.go:414] "containerMap: removing container" podUID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" containerName="maas-api"
Oct 31 00:06:46.092454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092444    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" containerName="maas-api"
Oct 31 00:06:46.092538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092530    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b7223597-3978-48e0-9914-d1b19cb1615b" containerName="maas-api"
Oct 31 00:06:46.092603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092594    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7223597-3978-48e0-9914-d1b19cb1615b" containerName="maas-api"
Oct 31 00:06:46.092849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092834    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.092931 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.092923    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa300c3-3d7a-4e3c-8fd0-fd8ec0a6d324" containerName="maas-api"
Oct 31 00:06:46.093054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.093023    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7223597-3978-48e0-9914-d1b19cb1615b" containerName="maas-api"
Oct 31 00:06:46.093188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.093175    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4ef6ff-fae1-4271-b157-3345c0fe108c" containerName="istio-proxy"
Oct 31 00:06:46.112845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.112810    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.243562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.243522    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-f7b8d8-openshift-default-dockercfg-gnkqm\""
Oct 31 00:06:46.268560 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.268521    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:06:46.304051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.303999    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304116    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304213    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7dt\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304264    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304287    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304395    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304430    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304491    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.304672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.304658    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.405742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405630    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.405742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405708    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405765    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7dt\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405812    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405835    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405921    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.405954    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406005    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406120    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406166    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406168    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406645    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.406747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.406702    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.407092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.407019    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.409245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.409218    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.409737 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.409711    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.465437 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.465404    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.465609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.465585    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7dt\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\") pod \"e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") " pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:46.563888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:46.563850    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:47.004051 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:06:47.003158    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:06:47.063084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.063050    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:06:47.067210 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:06:47.067179    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e5fd67_710d_46af_a46e_113e9177b836.slice/crio-fc8674e10877f18c466f76bf2eafe4b36f9ccc5b501de388a4984eb7bcf7d192 WatchSource:0}: Error finding container fc8674e10877f18c466f76bf2eafe4b36f9ccc5b501de388a4984eb7bcf7d192: Status 404 returned error can't find the container with id fc8674e10877f18c466f76bf2eafe4b36f9ccc5b501de388a4984eb7bcf7d192
Oct 31 00:06:47.069461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.069423    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:06:47.069571 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.069497    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:06:47.069571 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.069532    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:06:47.938191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.938147    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerStarted","Data":"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"}
Oct 31 00:06:47.938404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:47.938213    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerStarted","Data":"fc8674e10877f18c466f76bf2eafe4b36f9ccc5b501de388a4984eb7bcf7d192"}
Oct 31 00:06:48.564938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:48.564899    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:48.570807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:48.570773    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:48.686377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:48.686299    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podStartSLOduration=2.686281671 podStartE2EDuration="2.686281671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:48.150931071 +0000 UTC m=+1460.497914968" watchObservedRunningTime="2026-10-31 00:06:48.686281671 +0000 UTC m=+1461.033265569"
Oct 31 00:06:48.945667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:48.945577    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:48.946573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:48.946543    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:06:50.520586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:50.520546    6033 patch_prober.go:29] interesting pod/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:06:50.521023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:50.520602    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:06:50.965483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:50.965448    6033 generic.go:356] "Generic (PLEG): container finished" podID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerID="b1747b29f1f17f325323513f7d110bfd18dcff39c6e2451cb5c0b581c064712b" exitCode=0
Oct 31 00:06:50.965709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:50.965620    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerDied","Data":"b1747b29f1f17f325323513f7d110bfd18dcff39c6e2451cb5c0b581c064712b"}
Oct 31 00:06:50.965789 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:50.965720    6033 scope.go:122] "RemoveContainer" containerID="d1d830b22c87b0692ebf6fc2704d00e2932853d491682550c946887de3e74f8b"
Oct 31 00:06:51.107710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.107679    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:06:51.204305 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204269    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204331    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204431    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204515    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204605    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204684    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204781 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204721    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204781 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204743    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:51.204868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204800    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.204868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204859    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\") pod \"c7445965-9cd7-489c-aa9d-14b624f119cb\" (UID: \"c7445965-9cd7-489c-aa9d-14b624f119cb\") "
Oct 31 00:06:51.205079 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.204995    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:51.205177 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205161    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:51.205590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205336    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:51.205590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205442    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:51.205918 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205893    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.205961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205927    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.205961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205946    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7445965-9cd7-489c-aa9d-14b624f119cb-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.206044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205960    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.206044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.205974    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.206869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.206839    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "kube-api-access-vpvdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:51.207180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.207154    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:06:51.207271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.207160    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:06:51.207271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.207193    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token" pod "c7445965-9cd7-489c-aa9d-14b624f119cb" (UID: "c7445965-9cd7-489c-aa9d-14b624f119cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:51.307549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.307506    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.307549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.307546    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.307770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.307563    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvdh\" (UniqueName: \"kubernetes.io/projected/c7445965-9cd7-489c-aa9d-14b624f119cb-kube-api-access-vpvdh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.307770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.307577    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7445965-9cd7-489c-aa9d-14b624f119cb-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:06:51.974002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.973972    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"
Oct 31 00:06:51.974516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.974019    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc" event={"ID":"c7445965-9cd7-489c-aa9d-14b624f119cb","Type":"ContainerDied","Data":"cf08abee823077a0f8e6fc3f0eda7d82116ceba5205cd2688b3b332d45864dea"}
Oct 31 00:06:51.974516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:51.974092    6033 scope.go:122] "RemoveContainer" containerID="b1747b29f1f17f325323513f7d110bfd18dcff39c6e2451cb5c0b581c064712b"
Oct 31 00:06:52.183896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:52.183800    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:06:52.230236 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:52.230202    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d32a7cc0-gw-openshift-default-74cb955697-d8gkc"]
Oct 31 00:06:53.841902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:53.841867    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" path="/var/lib/kubelet/pods/c7445965-9cd7-489c-aa9d-14b624f119cb/volumes"
Oct 31 00:06:55.228480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.228445    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:06:55.229618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229591    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.229618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229617    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.229886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229652    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.229886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229659    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.229886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229829    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.229886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.229844    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7445965-9cd7-489c-aa9d-14b624f119cb" containerName="istio-proxy"
Oct 31 00:06:55.234673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.234645    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.289165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.289130    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:06:55.321563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.321506    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.321760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.321656    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjz8s\" (UniqueName: \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.330057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.330005    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-f7b8d8\""
Oct 31 00:06:55.376758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.376718    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-f7b8d8-dockercfg-5vzdt\""
Oct 31 00:06:55.423003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.422964    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.423218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.423168    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjz8s\" (UniqueName: \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.423782 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.423761    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.504981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.504942    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjz8s\" (UniqueName: \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\") pod \"payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.550987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.550948    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:55.961095 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:55.956473    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:06:55.961095 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:06:55.958324    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52733d7_dd08_435a_a1f7_7581e0657f89.slice/crio-c292dbbb120549482e7425b501dd96609f5f52545993277203d821a3780a211f WatchSource:0}: Error finding container c292dbbb120549482e7425b501dd96609f5f52545993277203d821a3780a211f: Status 404 returned error can't find the container with id c292dbbb120549482e7425b501dd96609f5f52545993277203d821a3780a211f
Oct 31 00:06:56.006635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:56.006598    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" event={"ID":"e52733d7-dd08-435a-a1f7-7581e0657f89","Type":"ContainerStarted","Data":"c292dbbb120549482e7425b501dd96609f5f52545993277203d821a3780a211f"}
Oct 31 00:06:57.016131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:57.016089    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" event={"ID":"e52733d7-dd08-435a-a1f7-7581e0657f89","Type":"ContainerStarted","Data":"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"}
Oct 31 00:06:57.016984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:57.016216    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:06:57.126406 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:06:57.126353    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" podStartSLOduration=2.12633925 podStartE2EDuration="2.12633925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:57.1261771 +0000 UTC m=+1469.473161025" watchObservedRunningTime="2026-10-31 00:06:57.12633925 +0000 UTC m=+1469.473323148"
Oct 31 00:07:00.182610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.182551    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:07:00.189143 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.189108    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.239970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.239690    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:07:00.281247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.280978    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-09efcb6d-serving-cert\""
Oct 31 00:07:00.303593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.303533    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.304561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.304512    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qhh\" (UniqueName: \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.304757 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.304738    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.307646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.307434    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.426021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.425985    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.426248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.426179    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.426341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.426269    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qhh\" (UniqueName: \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.426341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.426300    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.426594 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:00.426569    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-serving-cert: secret "maas-api-e2e-derive-09efcb6d-serving-cert" not found
Oct 31 00:07:00.426680 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:00.426655    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls podName:386d1eb2-7ddd-4e04-bdc0-3332520075af nodeName:}" failed. No retries permitted until 2026-10-31 00:07:00.926623054 +0000 UTC m=+1473.273606931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls") pod "maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" (UID: "386d1eb2-7ddd-4e04-bdc0-3332520075af") : secret "maas-api-e2e-derive-09efcb6d-serving-cert" not found
Oct 31 00:07:00.426892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.426859    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.429778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.429731    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.494156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.494020    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qhh\" (UniqueName: \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.936378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.936335    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:00.939290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:00.939260    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") pod \"maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:01.107937 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:01.107896    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:01.497100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:01.497063    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:07:01.498405 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:07:01.498361    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d1eb2_7ddd_4e04_bdc0_3332520075af.slice/crio-6493dbdf960cf42e6f4b3e12935109ef3c5728c8fb78f3041dfd5967847be56b WatchSource:0}: Error finding container 6493dbdf960cf42e6f4b3e12935109ef3c5728c8fb78f3041dfd5967847be56b: Status 404 returned error can't find the container with id 6493dbdf960cf42e6f4b3e12935109ef3c5728c8fb78f3041dfd5967847be56b
Oct 31 00:07:02.059373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:02.059323    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" event={"ID":"386d1eb2-7ddd-4e04-bdc0-3332520075af","Type":"ContainerStarted","Data":"6493dbdf960cf42e6f4b3e12935109ef3c5728c8fb78f3041dfd5967847be56b"}
Oct 31 00:07:03.068420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:03.068382    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" event={"ID":"386d1eb2-7ddd-4e04-bdc0-3332520075af","Type":"ContainerStarted","Data":"248ea95bb93605b453b33e4e9d2597de4880a517336c8caaa39bc9555b75968d"}
Oct 31 00:07:03.068895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:03.068475    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:03.180460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:03.180407    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" podStartSLOduration=2.10272148 podStartE2EDuration="3.180391844s" totalImagesPullingTime="1.077670364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:03.179329465 +0000 UTC m=+1475.526313354" watchObservedRunningTime="2026-10-31 00:07:03.180391844 +0000 UTC m=+1475.527375741"
Oct 31 00:07:08.027022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:08.026979    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:07:09.105267 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:09.105233    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:07:11.659443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:11.659392    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:07:11.710237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:11.710180    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:07:12.698053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:12.697974    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:07:12.698684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:12.698083    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:07:16.972758 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:16.972716    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:07:18.027796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:18.027758    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:07:18.028295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:18.027976    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy" containerID="cri-o://c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9" gracePeriod=30
Oct 31 00:07:23.307845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.307773    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:07:23.452315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452280    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452321    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452352    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452466    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452552    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452572    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452631    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452682    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.452875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452771    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:23.452875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.452821    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\") pod \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\" (UID: \"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06\") "
Oct 31 00:07:23.453290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453262    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:23.453409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453290    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:23.453478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453452    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:23.453478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453467    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:23.453818 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453797    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.453946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453821    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.453946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453834    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.453946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453849    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.453946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.453862    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.455509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.455476    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:07:23.455509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.455484    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:23.468933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.468889    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:23.468933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.468944    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b" pod "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" (UID: "4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06"). InnerVolumeSpecName "kube-api-access-gvv9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:23.555235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.555192    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.555235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.555232    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.555457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.555247    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvv9b\" (UniqueName: \"kubernetes.io/projected/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-kube-api-access-gvv9b\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.555457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.555261    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:23.595098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.594997    6033 generic.go:356] "Generic (PLEG): container finished" podID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerID="c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9" exitCode=0
Oct 31 00:07:23.595098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.595085    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"
Oct 31 00:07:23.595283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.595088    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" event={"ID":"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06","Type":"ContainerDied","Data":"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"}
Oct 31 00:07:23.595283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.595187    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl" event={"ID":"4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06","Type":"ContainerDied","Data":"867a2cbef655f39225d641e994b9f0cf93d64ee7771341afd3c4efe41d0081f2"}
Oct 31 00:07:23.595283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.595212    6033 scope.go:122] "RemoveContainer" containerID="c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"
Oct 31 00:07:23.607647 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.607626    6033 scope.go:122] "RemoveContainer" containerID="c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"
Oct 31 00:07:23.608024 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:23.608003    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9\": container with ID starting with c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9 not found: ID does not exist" containerID="c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"
Oct 31 00:07:23.608103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.608076    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9"} err="failed to get container status \"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9\": rpc error: code = NotFound desc = could not find container \"c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9\": container with ID starting with c6f4eb63f28d564d6a3d44ea4c7040d45f02b31369aa01d0c11c15519a272ec9 not found: ID does not exist"
Oct 31 00:07:23.752088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.752044    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:07:23.756041 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.755990    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-n4hsl"]
Oct 31 00:07:23.831861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:23.831824    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" path="/var/lib/kubelet/pods/4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06/volumes"
Oct 31 00:07:26.714063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:26.714015    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:07:30.862488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:30.862451    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:07:31.714907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:31.714854    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" podUID="b4290c22-faab-475c-8464-574daff9caf9" containerName="payload-processing" containerID="cri-o://282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445" gracePeriod=25
Oct 31 00:07:32.032211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.032173    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:07:32.056574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.056529    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"]
Oct 31 00:07:32.103338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.103306    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aee852-rn8t5"]
Oct 31 00:07:32.148527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.148493    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\" (UniqueName: \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\") pod \"b4290c22-faab-475c-8464-574daff9caf9\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") "
Oct 31 00:07:32.148748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.148614    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\") pod \"b4290c22-faab-475c-8464-574daff9caf9\" (UID: \"b4290c22-faab-475c-8464-574daff9caf9\") "
Oct 31 00:07:32.148981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.148955    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume" pod "b4290c22-faab-475c-8464-574daff9caf9" (UID: "b4290c22-faab-475c-8464-574daff9caf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:32.150777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.150746    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb" pod "b4290c22-faab-475c-8464-574daff9caf9" (UID: "b4290c22-faab-475c-8464-574daff9caf9"). InnerVolumeSpecName "kube-api-access-phqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:32.250118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.250066    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4290c22-faab-475c-8464-574daff9caf9-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:32.250118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.250126    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phqwb\" (UniqueName: \"kubernetes.io/projected/b4290c22-faab-475c-8464-574daff9caf9-kube-api-access-phqwb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:07:32.665629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.665480    6033 generic.go:356] "Generic (PLEG): container finished" podID="b4290c22-faab-475c-8464-574daff9caf9" containerID="282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445" exitCode=0
Oct 31 00:07:32.665629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.665532    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"
Oct 31 00:07:32.665629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.665559    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" event={"ID":"b4290c22-faab-475c-8464-574daff9caf9","Type":"ContainerDied","Data":"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"}
Oct 31 00:07:32.665949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.665631    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x" event={"ID":"b4290c22-faab-475c-8464-574daff9caf9","Type":"ContainerDied","Data":"37dc191708813cea9cc74b17e8ac010a8994c210d4383744cd954b1acaed9f08"}
Oct 31 00:07:32.665949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.665664    6033 scope.go:122] "RemoveContainer" containerID="282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"
Oct 31 00:07:32.681972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.681942    6033 scope.go:122] "RemoveContainer" containerID="282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"
Oct 31 00:07:32.684090 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:32.683348    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445\": container with ID starting with 282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445 not found: ID does not exist" containerID="282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"
Oct 31 00:07:32.684090 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.683602    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445"} err="failed to get container status \"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445\": rpc error: code = NotFound desc = could not find container \"282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445\": container with ID starting with 282a3fcbe935fe78e2baadf3d0d515d1dfd37fab4036a0c6ba94aecf44a96445 not found: ID does not exist"
Oct 31 00:07:32.824342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.824305    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:07:32.871441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:32.871398    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e0751b65-769966b6c4-9pw2x"]
Oct 31 00:07:33.832767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:33.832728    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8948ba50-ab14-4f89-a476-2a9921154fdf" path="/var/lib/kubelet/pods/8948ba50-ab14-4f89-a476-2a9921154fdf/volumes"
Oct 31 00:07:33.833293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:33.833275    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4290c22-faab-475c-8464-574daff9caf9" path="/var/lib/kubelet/pods/b4290c22-faab-475c-8464-574daff9caf9/volumes"
Oct 31 00:07:43.722867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.722824    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:07:43.723587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723569    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b4290c22-faab-475c-8464-574daff9caf9" containerName="payload-processing"
Oct 31 00:07:43.723587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723589    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4290c22-faab-475c-8464-574daff9caf9" containerName="payload-processing"
Oct 31 00:07:43.723712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723605    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy"
Oct 31 00:07:43.723712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723612    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy"
Oct 31 00:07:43.723712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723709    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4290c22-faab-475c-8464-574daff9caf9" containerName="payload-processing"
Oct 31 00:07:43.723807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.723722    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c92dfc0-58e8-4ac1-9dbf-4c93b379fd06" containerName="istio-proxy"
Oct 31 00:07:43.727734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.727705    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.778547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.778462    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-f7b8d8-serving-cert\""
Oct 31 00:07:43.830329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.830286    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8s8\" (UniqueName: \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.830634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.830605    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.830699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.830677    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.830760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.830739    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.835107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.835079    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:07:43.931766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.931718    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.931766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.931770    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.932022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.931819    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.932022 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:43.931870    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-serving-cert: secret "maas-api-e2e-shared-a-w6-f7b8d8-serving-cert" not found
Oct 31 00:07:43.932022 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:43.931947    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls podName:ec19024f-267e-48a7-9589-960c9892d889 nodeName:}" failed. No retries permitted until 2026-10-31 00:07:44.431925199 +0000 UTC m=+1516.778909095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls") pod "maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" (UID: "ec19024f-267e-48a7-9589-960c9892d889") : secret "maas-api-e2e-shared-a-w6-f7b8d8-serving-cert" not found
Oct 31 00:07:43.932022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.932006    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8s8\" (UniqueName: \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.932581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.932560    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.934219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.934201    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:43.990895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:43.990819    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8s8\" (UniqueName: \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:44.441325 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:44.441279    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:44.443893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:44.443862    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:44.641987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:44.641942    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:45.017238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:45.017198    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:07:45.018137 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:07:45.018107    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec19024f_267e_48a7_9589_960c9892d889.slice/crio-13c91160fc44926adcee46486761feb749e71bb7c58ac8291ee0064cfd3d0d6d WatchSource:0}: Error finding container 13c91160fc44926adcee46486761feb749e71bb7c58ac8291ee0064cfd3d0d6d: Status 404 returned error can't find the container with id 13c91160fc44926adcee46486761feb749e71bb7c58ac8291ee0064cfd3d0d6d
Oct 31 00:07:45.785294 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:45.785214    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" event={"ID":"ec19024f-267e-48a7-9589-960c9892d889","Type":"ContainerStarted","Data":"13c91160fc44926adcee46486761feb749e71bb7c58ac8291ee0064cfd3d0d6d"}
Oct 31 00:07:46.796864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:46.796821    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" event={"ID":"ec19024f-267e-48a7-9589-960c9892d889","Type":"ContainerStarted","Data":"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"}
Oct 31 00:07:46.797346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:46.797224    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:46.909837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:46.909771    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" podStartSLOduration=2.987607847 podStartE2EDuration="3.909756008s" totalImagesPullingTime="922.148161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:46.907858745 +0000 UTC m=+1519.254842644" watchObservedRunningTime="2026-10-31 00:07:46.909756008 +0000 UTC m=+1519.256739906"
Oct 31 00:07:46.972594 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:07:46.972556    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:07:52.809808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:52.809772    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:07:53.060136 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:53.060043    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"]
Oct 31 00:07:53.063338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:53.063305    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-473a7013-k2qxg"]
Oct 31 00:07:53.832351 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:07:53.832315    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5e9846-3db5-4e84-af64-c89fe2b16636" path="/var/lib/kubelet/pods/df5e9846-3db5-4e84-af64-c89fe2b16636/volumes"
Oct 31 00:08:02.083969 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.083928    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:08:02.090456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.090428    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.131373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.131336    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcw9\" (UniqueName: \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.131568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.131457    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.131568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.131483    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.131686 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.131657    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.140874 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.140837    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:08:02.183384 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.183333    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a251fdfb-serving-cert\""
Oct 31 00:08:02.232836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.232800    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcw9\" (UniqueName: \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.233108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.232878    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.233108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.232905    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.233108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.233000    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.233376 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:02.233224    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-serving-cert: secret "maas-api-e2e-mt-a251fdfb-serving-cert" not found
Oct 31 00:08:02.233376 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:02.233297    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls podName:ce44ee65-6516-4c15-9802-9e917636548d nodeName:}" failed. No retries permitted until 2026-10-31 00:08:02.733275559 +0000 UTC m=+1535.080259437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls") pod "maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" (UID: "ce44ee65-6516-4c15-9802-9e917636548d") : secret "maas-api-e2e-mt-a251fdfb-serving-cert" not found
Oct 31 00:08:02.233840 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.233809    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.235698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.235675    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.293185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.293146    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcw9\" (UniqueName: \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.741489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.741450    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:02.744314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:02.744279    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:03.006763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:03.006721    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:03.383926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:03.383889    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:08:03.385602 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:08:03.385574    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce44ee65_6516_4c15_9802_9e917636548d.slice/crio-d94a1eb60e79c0e5757cf3d4d2fec9e936d2eec59f6318abe7095e7980bde4cd WatchSource:0}: Error finding container d94a1eb60e79c0e5757cf3d4d2fec9e936d2eec59f6318abe7095e7980bde4cd: Status 404 returned error can't find the container with id d94a1eb60e79c0e5757cf3d4d2fec9e936d2eec59f6318abe7095e7980bde4cd
Oct 31 00:08:03.925686 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:03.925644    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" event={"ID":"ce44ee65-6516-4c15-9802-9e917636548d","Type":"ContainerStarted","Data":"d94a1eb60e79c0e5757cf3d4d2fec9e936d2eec59f6318abe7095e7980bde4cd"}
Oct 31 00:08:04.936204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:04.936160    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" event={"ID":"ce44ee65-6516-4c15-9802-9e917636548d","Type":"ContainerStarted","Data":"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"}
Oct 31 00:08:04.936632 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:04.936243    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:05.050341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:05.050279    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" podStartSLOduration=2.154129984 podStartE2EDuration="3.050260262s" totalImagesPullingTime="896.130278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:05.046823859 +0000 UTC m=+1537.393807758" watchObservedRunningTime="2026-10-31 00:08:05.050260262 +0000 UTC m=+1537.397244160"
Oct 31 00:08:10.047093 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:10.046958    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"]
Oct 31 00:08:10.093741 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:10.093704    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95470bd7-vtxhb"]
Oct 31 00:08:10.952824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:10.952792    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:08:11.832346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:11.832243    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54701d7-b389-465d-b274-f44903692999" path="/var/lib/kubelet/pods/b54701d7-b389-465d-b274-f44903692999/volumes"
Oct 31 00:08:16.526124 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:16.526084    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:08:16.973047 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:16.972942    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:08:19.046424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:19.046381    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"]
Oct 31 00:08:19.114165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:19.113876    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c3c974-m958w"]
Oct 31 00:08:19.834324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:19.834273    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac4db9f-7c88-43e9-89f3-4b843bf7050a" path="/var/lib/kubelet/pods/9ac4db9f-7c88-43e9-89f3-4b843bf7050a/volumes"
Oct 31 00:08:28.246343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.246309    6033 generic.go:356] "Generic (PLEG): container finished" podID="10e5fd67-710d-46af-a46e-113e9177b836" containerID="3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970" exitCode=0
Oct 31 00:08:28.246765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.246377    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerDied","Data":"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"}
Oct 31 00:08:28.246899 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.246884    6033 scope.go:122] "RemoveContainer" containerID="3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"
Oct 31 00:08:28.248621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.248585    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:08:28.248754 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.248673    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:08:28.248754 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:28.248721    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:08:29.256945 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:29.256896    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerStarted","Data":"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"}
Oct 31 00:08:29.564230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:29.564137    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:08:29.565733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:29.565706    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:29.565862 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:29.565768    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:30.564914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:30.564870    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:30.565406 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:30.564936    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:31.432694 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:31.432649    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:08:31.433918 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:31.433259    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy" containerID="cri-o://f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d" gracePeriod=30
Oct 31 00:08:31.564357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:31.564317    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:31.564545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:31.564375    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:32.564475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:32.564429    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:32.565014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:32.564508    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:33.564633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:33.564598    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:33.565107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:33.564655    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:33.946325 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:33.946227    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:08:34.564751 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:34.564716    6033 patch_prober.go:29] interesting pod/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 00:08:34.565185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:34.564780    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 00:08:35.570251 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:35.570211    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:08:35.571526 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:35.571498    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:08:36.724214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.724185    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:08:36.838425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838386    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838522    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838553    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838634    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838701    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838733    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838781    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:08:36.838847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838834    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838879    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.838956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838939    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\") pod \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\" (UID: \"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb\") "
Oct 31 00:08:36.839135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.838974    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:36.839217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839173    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:36.839217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839188    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:36.839521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839497    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:36.839850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839827    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.839926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839856    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.839926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839875    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.839926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839888    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.839926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.839902    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.841015 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.840985    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:08:36.841433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.841068    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "kube-api-access-wnrz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:36.841433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.841095    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:08:36.841433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.841239    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token" pod "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" (UID: "8b06a5d7-9314-4cc0-863a-01c1d10c0ebb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:36.942014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.941938    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.942014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.941973    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnrz5\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-kube-api-access-wnrz5\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.942014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.942008    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:36.942291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:36.942076    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:37.336807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.336772    6033 generic.go:356] "Generic (PLEG): container finished" podID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerID="f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d" exitCode=0
Oct 31 00:08:37.337078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.336823    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"
Oct 31 00:08:37.337078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.336857    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerDied","Data":"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"}
Oct 31 00:08:37.337078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.336920    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4" event={"ID":"8b06a5d7-9314-4cc0-863a-01c1d10c0ebb","Type":"ContainerDied","Data":"83fe5427a6ed7c4009e1a046f2ae2ecd765546218f5ad24f92337dadd1ebd797"}
Oct 31 00:08:37.337078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.336945    6033 scope.go:122] "RemoveContainer" containerID="f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"
Oct 31 00:08:37.349094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.349069    6033 scope.go:122] "RemoveContainer" containerID="0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"
Oct 31 00:08:37.361092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.361071    6033 scope.go:122] "RemoveContainer" containerID="f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"
Oct 31 00:08:37.361478 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:37.361457    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d\": container with ID starting with f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d not found: ID does not exist" containerID="f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"
Oct 31 00:08:37.361552 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.361494    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d"} err="failed to get container status \"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d\": rpc error: code = NotFound desc = could not find container \"f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d\": container with ID starting with f8af48239a47c34878ee3f7ce8b0f088e050822d541e000eedf02b7cc5528e2d not found: ID does not exist"
Oct 31 00:08:37.361552 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.361525    6033 scope.go:122] "RemoveContainer" containerID="0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"
Oct 31 00:08:37.361840 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:37.361821    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96\": container with ID starting with 0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96 not found: ID does not exist" containerID="0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"
Oct 31 00:08:37.361913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.361844    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96"} err="failed to get container status \"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96\": rpc error: code = NotFound desc = could not find container \"0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96\": container with ID starting with 0ba59f9c2eb24c1047a897a3f669f422a92ae9a42e3fcae2aa8c2e8638a80a96 not found: ID does not exist"
Oct 31 00:08:37.496633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.496596    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:08:37.543557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.543516    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e0751b65-openshift-default-5846888d89-2pzt4"]
Oct 31 00:08:37.833238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:37.833198    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" path="/var/lib/kubelet/pods/8b06a5d7-9314-4cc0-863a-01c1d10c0ebb/volumes"
Oct 31 00:08:46.975594 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:08:46.975558    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:08:48.301698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.301657    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:08:48.302813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.302782    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:08:48.302813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.302807    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:08:48.302979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.302864    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:08:48.302979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.302872    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:08:48.303072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.303013    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:08:48.306094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.306018    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.405494 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.405461    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-71b76a-dockercfg-9skgj\""
Oct 31 00:08:48.415703 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.415672    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71b76a\""
Oct 31 00:08:48.483066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.480822    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.483066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.480988    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfbn\" (UniqueName: \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.517922 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.517891    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:08:48.582501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.582400    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.582660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.582507    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfbn\" (UniqueName: \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.583301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.583278    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.643882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.643846    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfbn\" (UniqueName: \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\") pod \"payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") " pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:48.923556 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:48.923429    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:49.362550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:49.362516    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:08:49.365393 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:08:49.365353    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a6369d_d2f2_4a63_ab41_e42116774e76.slice/crio-3ea46bf5dfb7abe4ac71a0ac0ab231bc0cdf839ecc444c0b961cecf29b6314a6 WatchSource:0}: Error finding container 3ea46bf5dfb7abe4ac71a0ac0ab231bc0cdf839ecc444c0b961cecf29b6314a6: Status 404 returned error can't find the container with id 3ea46bf5dfb7abe4ac71a0ac0ab231bc0cdf839ecc444c0b961cecf29b6314a6
Oct 31 00:08:49.543657 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:49.543598    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" event={"ID":"d4a6369d-d2f2-4a63-ab41-e42116774e76","Type":"ContainerStarted","Data":"05e2683f4df29d7f2ca123d888bfdd456d9e4b6f3d0ca9b78b2ba6d00c458f82"}
Oct 31 00:08:49.543878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:49.543688    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" event={"ID":"d4a6369d-d2f2-4a63-ab41-e42116774e76","Type":"ContainerStarted","Data":"3ea46bf5dfb7abe4ac71a0ac0ab231bc0cdf839ecc444c0b961cecf29b6314a6"}
Oct 31 00:08:49.543878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:49.543759    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:08:49.653525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:49.653099    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" podStartSLOduration=1.653078013 podStartE2EDuration="1.653078013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:49.652658549 +0000 UTC m=+1581.999642474" watchObservedRunningTime="2026-10-31 00:08:49.653078013 +0000 UTC m=+1582.000061912"
Oct 31 00:08:57.410319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.410221    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:08:57.411182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.411143    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" podUID="386d1eb2-7ddd-4e04-bdc0-3332520075af" containerName="maas-api" containerID="cri-o://248ea95bb93605b453b33e4e9d2597de4880a517336c8caaa39bc9555b75968d" gracePeriod=30
Oct 31 00:08:57.640525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.640253    6033 generic.go:356] "Generic (PLEG): container finished" podID="386d1eb2-7ddd-4e04-bdc0-3332520075af" containerID="248ea95bb93605b453b33e4e9d2597de4880a517336c8caaa39bc9555b75968d" exitCode=0
Oct 31 00:08:57.640525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.640429    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" event={"ID":"386d1eb2-7ddd-4e04-bdc0-3332520075af","Type":"ContainerDied","Data":"248ea95bb93605b453b33e4e9d2597de4880a517336c8caaa39bc9555b75968d"}
Oct 31 00:08:57.836989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.836949    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:08:57.995221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.995186    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\" (UniqueName: \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\") pod \"386d1eb2-7ddd-4e04-bdc0-3332520075af\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") "
Oct 31 00:08:57.995412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.995260    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\") pod \"386d1eb2-7ddd-4e04-bdc0-3332520075af\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") "
Oct 31 00:08:57.995412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.995369    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") pod \"386d1eb2-7ddd-4e04-bdc0-3332520075af\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") "
Oct 31 00:08:57.995597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.995563    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\") pod \"386d1eb2-7ddd-4e04-bdc0-3332520075af\" (UID: \"386d1eb2-7ddd-4e04-bdc0-3332520075af\") "
Oct 31 00:08:57.998021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:57.997986    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca" pod "386d1eb2-7ddd-4e04-bdc0-3332520075af" (UID: "386d1eb2-7ddd-4e04-bdc0-3332520075af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:08:58.001483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.001090    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh" pod "386d1eb2-7ddd-4e04-bdc0-3332520075af" (UID: "386d1eb2-7ddd-4e04-bdc0-3332520075af"). InnerVolumeSpecName "kube-api-access-f7qhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:08:58.005536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.005492    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls" pod "386d1eb2-7ddd-4e04-bdc0-3332520075af" (UID: "386d1eb2-7ddd-4e04-bdc0-3332520075af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:08:58.009419 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.009024    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs" pod "386d1eb2-7ddd-4e04-bdc0-3332520075af" (UID: "386d1eb2-7ddd-4e04-bdc0-3332520075af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:08:58.103771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.103667    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qhh\" (UniqueName: \"kubernetes.io/projected/386d1eb2-7ddd-4e04-bdc0-3332520075af-kube-api-access-f7qhh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:58.103771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.103715    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386d1eb2-7ddd-4e04-bdc0-3332520075af-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:58.103771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.103733    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:58.103771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.103744    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386d1eb2-7ddd-4e04-bdc0-3332520075af-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:08:58.656784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.656750    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"
Oct 31 00:08:58.657427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.656784    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq" event={"ID":"386d1eb2-7ddd-4e04-bdc0-3332520075af","Type":"ContainerDied","Data":"6493dbdf960cf42e6f4b3e12935109ef3c5728c8fb78f3041dfd5967847be56b"}
Oct 31 00:08:58.657427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.656869    6033 scope.go:122] "RemoveContainer" containerID="248ea95bb93605b453b33e4e9d2597de4880a517336c8caaa39bc9555b75968d"
Oct 31 00:08:58.813674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.813635    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:08:58.817324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:58.817293    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-09efcb6d-77c9f69575-mxfvq"]
Oct 31 00:08:59.843045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:08:59.843000    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386d1eb2-7ddd-4e04-bdc0-3332520075af" path="/var/lib/kubelet/pods/386d1eb2-7ddd-4e04-bdc0-3332520075af/volumes"
Oct 31 00:09:00.560727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:00.560698    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:09:16.971808 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:09:16.971769    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:09:28.224088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:28.224052    6033 scope.go:122] "RemoveContainer" containerID="fb5cb826b7415eae18d6304da2789f4826002bfab63334506da3abac38e7812b"
Oct 31 00:09:28.235103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:28.235074    6033 scope.go:122] "RemoveContainer" containerID="8431d3eeb886fa7618f25a9a7bb21531702f4aa7535383ed23cb407b2d984fd5"
Oct 31 00:09:28.245445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:28.245422    6033 scope.go:122] "RemoveContainer" containerID="a05cfd74a1388da1d700e585e311d477293d4d04fa7eb506a648d0bd8a3b2bdc"
Oct 31 00:09:28.255726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:28.255704    6033 scope.go:122] "RemoveContainer" containerID="21454803f65d334e9626c046bfa7b64d9ba20b24b22da9f6913ac5b5448ca742"
Oct 31 00:09:30.159104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.159019    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:09:30.159797 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.159780    6033 cpu_manager.go:414] "containerMap: removing container" podUID="386d1eb2-7ddd-4e04-bdc0-3332520075af" containerName="maas-api"
Oct 31 00:09:30.159843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.159798    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386d1eb2-7ddd-4e04-bdc0-3332520075af" containerName="maas-api"
Oct 31 00:09:30.159948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.159939    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b06a5d7-9314-4cc0-863a-01c1d10c0ebb" containerName="istio-proxy"
Oct 31 00:09:30.159986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.159961    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="386d1eb2-7ddd-4e04-bdc0-3332520075af" containerName="maas-api"
Oct 31 00:09:30.164947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.164921    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.214210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.214169    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:09:30.214210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.214169    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:09:30.214454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.214304    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:09:30.214454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.214348    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbdd5\""
Oct 31 00:09:30.267442 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267402    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.267636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267450    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npw8p\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.267636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267543    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.267765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267745    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.267802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267787    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.267925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.267906    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.268023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.268011    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.361483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.361431    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:09:30.370157 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370105    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370181    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370290    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370408    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370484 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370452    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370520 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370490    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npw8p\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.370561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.370523    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.371688 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.371658    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.373881 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.373852    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.375673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.375643    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.375861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.375685    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.376123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.376096    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.486551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.486456    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.488857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.488761    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npw8p\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzzph\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:30.779163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:30.779129    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:31.208581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:31.208545    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:09:31.210257 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:09:31.210218    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355ca3df_4302_4a5f_8712_f4299e910319.slice/crio-fae8bfd6a7046345b1eac06849b965304d44816e850ac8c25cd7f18a6b2121ba WatchSource:0}: Error finding container fae8bfd6a7046345b1eac06849b965304d44816e850ac8c25cd7f18a6b2121ba: Status 404 returned error can't find the container with id fae8bfd6a7046345b1eac06849b965304d44816e850ac8c25cd7f18a6b2121ba
Oct 31 00:09:31.212634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:31.212614    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:09:32.014351 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:32.014312    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" event={"ID":"355ca3df-4302-4a5f-8712-f4299e910319","Type":"ContainerStarted","Data":"fae8bfd6a7046345b1eac06849b965304d44816e850ac8c25cd7f18a6b2121ba"}
Oct 31 00:09:32.667603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:32.667559    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:09:32.668192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:32.667790    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy" containerID="cri-o://a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771" gracePeriod=30
Oct 31 00:09:35.348263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:35.348218    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:09:35.348612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:35.348285    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:09:36.049276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.049232    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" event={"ID":"355ca3df-4302-4a5f-8712-f4299e910319","Type":"ContainerStarted","Data":"53ca2a6906d9dc5c0ab979b0232510e7e10cf63fea1fb29aa243728f0a30430d"}
Oct 31 00:09:36.049691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.049661    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:36.051860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.051792    6033 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzzph container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:09:36.051860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.051842    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:09:36.168341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.168281    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" podStartSLOduration=2.034453738 podStartE2EDuration="6.168262007s" totalImagesPullingTime="4.133808269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:36.163073337 +0000 UTC m=+1628.510057235" watchObservedRunningTime="2026-10-31 00:09:36.168262007 +0000 UTC m=+1628.515245905"
Oct 31 00:09:36.930055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:36.926546    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:09:37.057857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:37.057821    6033 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzzph container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:09:37.058104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:37.057899    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:09:38.050046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.050008    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:09:38.065460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.065426    6033 generic.go:356] "Generic (PLEG): container finished" podID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerID="a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771" exitCode=0
Oct 31 00:09:38.065636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.065487    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"
Oct 31 00:09:38.065636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.065507    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" event={"ID":"3e0e7bdf-b088-47c0-8e28-3fb78307b77d","Type":"ContainerDied","Data":"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"}
Oct 31 00:09:38.065636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.065578    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh" event={"ID":"3e0e7bdf-b088-47c0-8e28-3fb78307b77d","Type":"ContainerDied","Data":"d9b9b94f3ea9f2c3fe6c10ea43210736e936b458acc588438e0a9bb0aa32ab0b"}
Oct 31 00:09:38.065636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.065612    6033 scope.go:122] "RemoveContainer" containerID="a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"
Oct 31 00:09:38.076961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.076940    6033 scope.go:122] "RemoveContainer" containerID="a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"
Oct 31 00:09:38.077492 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:09:38.077463    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771\": container with ID starting with a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771 not found: ID does not exist" containerID="a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"
Oct 31 00:09:38.077613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.077504    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771"} err="failed to get container status \"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771\": rpc error: code = NotFound desc = could not find container \"a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771\": container with ID starting with a3cc58eae875c02ee6cf94c28f1c5d43f0232df7ed524597264d8a7cb31dc771 not found: ID does not exist"
Oct 31 00:09:38.122286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122234    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122303    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122338    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122455    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122491    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122577    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122585    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.122669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122646    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122764    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.122897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122779    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.122897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122817    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\") pod \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\" (UID: \"3e0e7bdf-b088-47c0-8e28-3fb78307b77d\") "
Oct 31 00:09:38.123111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.122968    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.123645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.123619    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.123645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.123646    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.123885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.123661    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.124013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.123994    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:09:38.124361 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.124335    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.126201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.126175    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:09:38.126343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.126174    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "kube-api-access-kh499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:38.126409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.126336    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:09:38.127414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.127394    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token" pod "3e0e7bdf-b088-47c0-8e28-3fb78307b77d" (UID: "3e0e7bdf-b088-47c0-8e28-3fb78307b77d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225227    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225260    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225270    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225282    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225296    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.225319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.225312    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh499\" (UniqueName: \"kubernetes.io/projected/3e0e7bdf-b088-47c0-8e28-3fb78307b77d-kube-api-access-kh499\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:09:38.491072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.490977    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:09:38.496622 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:38.496588    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-4strh"]
Oct 31 00:09:39.831217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:39.831173    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" path="/var/lib/kubelet/pods/3e0e7bdf-b088-47c0-8e28-3fb78307b77d/volumes"
Oct 31 00:09:40.057530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:09:40.057491    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:09:46.971862 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:09:46.971821    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:10:16.972268 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:10:16.972224    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:10:27.042437 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:10:27.042403    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:10:46.972942 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:10:46.972904    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:11:10.533275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:10.533181    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:11:16.972964 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:16.972923    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:11:33.725048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:33.724983    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:11:33.725647 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:33.725285    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" containerID="cri-o://86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c" gracePeriod=30
Oct 31 00:11:33.772195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:33.772147    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 31 00:11:33.772443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:33.772409    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" containerID="cri-o://b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9" gracePeriod=30
Oct 31 00:11:34.954185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:34.954141    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 31 00:11:34.954713 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:34.954411    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" containerID="cri-o://92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720" gracePeriod=30
Oct 31 00:11:37.272383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:37.272343    6033 patch_prober.go:29] interesting pod/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:11:37.272796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:37.272406    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:11:37.916805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:37.916765    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:11:37.916986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:37.916845    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:11:38.798967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:38.798934    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 00:11:38.799381 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:38.798997    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 00:11:39.079845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.079808    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:11:39.113443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.113418    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 31 00:11:39.207966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.207934    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208017    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208113    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208160    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208234    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208268    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208286    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208291    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.208352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208321    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.208588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208372    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.208588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208526    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.208588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208548    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\") pod \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\" (UID: \"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5\") "
Oct 31 00:11:39.209005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.208578    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.209182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209114    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:39.209562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209535    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.209562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209564    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.209764 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209582    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.209764 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209595    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.209764 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.209607    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.211312 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.211286    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.211434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.211314    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "kube-api-access-wpxzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.212740 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.212712    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:39.212900 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.212883    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy" pod "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" (UID: "8d8ba6a3-5c8b-403b-a452-982ae35ad9d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.310813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.310724    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.310979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.310843    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.310979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.310894    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311118    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311269 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311251    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311259    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.311352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311288    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311345    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311362    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.311524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311373    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311443    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\") pod \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\" (UID: \"392b5af8-5042-4f20-a6dc-8d4ac3a62c81\") "
Oct 31 00:11:39.311657 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311585    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.311709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311684    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:39.311794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.311775    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.312341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312318    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312344    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312360    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312372    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312384    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312396    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312407    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312420    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxzb\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-kube-api-access-wpxzb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.312475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.312433    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.314281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314251    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.314396 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314275    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "kube-api-access-s8rnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:39.314396 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314302    6033 generic.go:356] "Generic (PLEG): container finished" podID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerID="86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c" exitCode=0
Oct 31 00:11:39.314396 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314364    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"
Oct 31 00:11:39.314547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314389    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" event={"ID":"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5","Type":"ContainerDied","Data":"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"}
Oct 31 00:11:39.314547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314440    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:39.314547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314451    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr" event={"ID":"8d8ba6a3-5c8b-403b-a452-982ae35ad9d5","Type":"ContainerDied","Data":"186bc1e590a2f26a6ce35bb8ec6c25711fd42e659e9133773183f3997901c4dc"}
Oct 31 00:11:39.314547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314474    6033 scope.go:122] "RemoveContainer" containerID="86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"
Oct 31 00:11:39.314680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.314469    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo" pod "392b5af8-5042-4f20-a6dc-8d4ac3a62c81" (UID: "392b5af8-5042-4f20-a6dc-8d4ac3a62c81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:39.316418 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.316394    6033 generic.go:356] "Generic (PLEG): container finished" podID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerID="b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9" exitCode=0
Oct 31 00:11:39.316510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.316501    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" event={"ID":"392b5af8-5042-4f20-a6dc-8d4ac3a62c81","Type":"ContainerDied","Data":"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"}
Oct 31 00:11:39.316553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.316538    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx" event={"ID":"392b5af8-5042-4f20-a6dc-8d4ac3a62c81","Type":"ContainerDied","Data":"0c09d045cd9b7c904824b78b7595bf2455ab130dea86fd7bca7db4e79cf9e223"}
Oct 31 00:11:39.316601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.316591    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"
Oct 31 00:11:39.330455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.330434    6033 scope.go:122] "RemoveContainer" containerID="86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"
Oct 31 00:11:39.330815 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:39.330792    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c\": container with ID starting with 86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c not found: ID does not exist" containerID="86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"
Oct 31 00:11:39.330925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.330825    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c"} err="failed to get container status \"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c\": rpc error: code = NotFound desc = could not find container \"86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c\": container with ID starting with 86c1e3bf4d2496a4be81df05986b23319603006abad978936d81f09e7d919b4c not found: ID does not exist"
Oct 31 00:11:39.330925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.330846    6033 scope.go:122] "RemoveContainer" containerID="b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"
Oct 31 00:11:39.343984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.343958    6033 scope.go:122] "RemoveContainer" containerID="b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"
Oct 31 00:11:39.344354 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:39.344335    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9\": container with ID starting with b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9 not found: ID does not exist" containerID="b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"
Oct 31 00:11:39.344440 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.344362    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9"} err="failed to get container status \"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9\": rpc error: code = NotFound desc = could not find container \"b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9\": container with ID starting with b91d421d3de6e28ff99ea99f9e3ef6fdfbe3fb4b3781864507e5e5866d757dd9 not found: ID does not exist"
Oct 31 00:11:39.413373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.413337    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.413373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.413367    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.413373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.413379    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.413633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.413389    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rnf\" (UniqueName: \"kubernetes.io/projected/392b5af8-5042-4f20-a6dc-8d4ac3a62c81-kube-api-access-s8rnf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:39.466117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.466073    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:11:39.470831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.470795    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-x59pr"]
Oct 31 00:11:39.574163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.574045    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 31 00:11:39.578268 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.578237    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-4jrqx"]
Oct 31 00:11:39.832281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.832200    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" path="/var/lib/kubelet/pods/392b5af8-5042-4f20-a6dc-8d4ac3a62c81/volumes"
Oct 31 00:11:39.832748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:39.832733    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" path="/var/lib/kubelet/pods/8d8ba6a3-5c8b-403b-a452-982ae35ad9d5/volumes"
Oct 31 00:11:40.126426 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:40.126334    6033 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37b9ae7_6ac7_43e4_8157_84545067dd7f.slice/crio-92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37b9ae7_6ac7_43e4_8157_84545067dd7f.slice/crio-conmon-92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:11:40.306055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.302744    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:11:40.381398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.380538    6033 generic.go:356] "Generic (PLEG): container finished" podID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerID="92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720" exitCode=0
Oct 31 00:11:40.381398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.380785    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" event={"ID":"e37b9ae7-6ac7-43e4-8157-84545067dd7f","Type":"ContainerDied","Data":"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"}
Oct 31 00:11:40.381398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.380881    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt" event={"ID":"e37b9ae7-6ac7-43e4-8157-84545067dd7f","Type":"ContainerDied","Data":"962795fde54b91c79e5d40ede7ac2e18246928ba4e33b3fad006e4e6bcd1adc2"}
Oct 31 00:11:40.381398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.380914    6033 scope.go:122] "RemoveContainer" containerID="92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"
Oct 31 00:11:40.381398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.381095    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"
Oct 31 00:11:40.400728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.400700    6033 scope.go:122] "RemoveContainer" containerID="92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"
Oct 31 00:11:40.401235 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:40.401203    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720\": container with ID starting with 92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720 not found: ID does not exist" containerID="92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"
Oct 31 00:11:40.401330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.401247    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720"} err="failed to get container status \"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720\": rpc error: code = NotFound desc = could not find container \"92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720\": container with ID starting with 92b30a03d1af0f7d4bdb6b033abebc0eb3db664d48be5990cb43d4738e0a1720 not found: ID does not exist"
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.440890    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441142    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441300    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441406    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441482    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441594    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441633    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441688    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.441748    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\") pod \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\" (UID: \"e37b9ae7-6ac7-43e4-8157-84545067dd7f\") "
Oct 31 00:11:40.446049 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.443716    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:40.450922 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.450420    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:40.450922 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.450624    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:40.451798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.451771    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:40.451998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.451946    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:40.463422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.463246    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:40.465714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.465665    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:40.466773 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.466402    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "kube-api-access-mz2s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:40.466773 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.466514    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token" pod "e37b9ae7-6ac7-43e4-8157-84545067dd7f" (UID: "e37b9ae7-6ac7-43e4-8157-84545067dd7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:40.545486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545439    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545487    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545509    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545525    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545539    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545553    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2s5\" (UniqueName: \"kubernetes.io/projected/e37b9ae7-6ac7-43e4-8157-84545067dd7f-kube-api-access-mz2s5\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545566    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545579    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.545780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.545596    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37b9ae7-6ac7-43e4-8157-84545067dd7f-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:11:40.825178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.825138    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 31 00:11:40.872265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:40.872218    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-thfjt"]
Oct 31 00:11:41.835490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:11:41.835453    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" path="/var/lib/kubelet/pods/e37b9ae7-6ac7-43e4-8157-84545067dd7f/volumes"
Oct 31 00:11:46.971855 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:11:46.971815    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:12:02.879200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879158    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"]
Oct 31 00:12:02.879823 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879808    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy"
Oct 31 00:12:02.879823 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879824    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879865    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879871    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879879    6033 cpu_manager.go:414] "containerMap: removing container" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879884    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879895    6033 cpu_manager.go:414] "containerMap: removing container" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy"
Oct 31 00:12:02.879910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.879899    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy"
Oct 31 00:12:02.880160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.880024    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8ba6a3-5c8b-403b-a452-982ae35ad9d5" containerName="istio-proxy"
Oct 31 00:12:02.880160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.880049    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0e7bdf-b088-47c0-8e28-3fb78307b77d" containerName="istio-proxy"
Oct 31 00:12:02.880160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.880060    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="392b5af8-5042-4f20-a6dc-8d4ac3a62c81" containerName="istio-proxy"
Oct 31 00:12:02.880160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.880075    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37b9ae7-6ac7-43e4-8157-84545067dd7f" containerName="istio-proxy"
Oct 31 00:12:02.883339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.883307    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:02.934096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.934019    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:12:02.980612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.980577    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:12:02.983723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.983694    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"]
Oct 31 00:12:02.988939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.988907    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:02.989111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.989094    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:02.989157 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:02.989128    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb25v\" (UniqueName: \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.090538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.090492    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.090748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.090699    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.090748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.090743    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb25v\" (UniqueName: \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.091633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.091599    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.093353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.093330    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.148609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.148527    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb25v\" (UniqueName: \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.198125 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.198087    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:03.601629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:03.601599    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"]
Oct 31 00:12:03.603562 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:12:03.603532    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646879f7_5a4d_49c8_93e2_26fabf23905f.slice/crio-f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f WatchSource:0}: Error finding container f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f: Status 404 returned error can't find the container with id f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f
Oct 31 00:12:04.589445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:04.589403    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m" event={"ID":"646879f7-5a4d-49c8-93e2-26fabf23905f","Type":"ContainerStarted","Data":"9a782b791a76cd78fa9575872c565cce2a0ee7fe19642dfb9bb122c45c17a4f2"}
Oct 31 00:12:04.589908 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:04.589465    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m" event={"ID":"646879f7-5a4d-49c8-93e2-26fabf23905f","Type":"ContainerStarted","Data":"f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f"}
Oct 31 00:12:04.700222 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:04.700160    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m" podStartSLOduration=2.7001436869999997 podStartE2EDuration="2.700143687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:04.698754589 +0000 UTC m=+1777.045738488" watchObservedRunningTime="2026-10-31 00:12:04.700143687 +0000 UTC m=+1777.047127585"
Oct 31 00:12:05.599477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:05.599440    6033 generic.go:356] "Generic (PLEG): container finished" podID="646879f7-5a4d-49c8-93e2-26fabf23905f" containerID="9a782b791a76cd78fa9575872c565cce2a0ee7fe19642dfb9bb122c45c17a4f2" exitCode=0
Oct 31 00:12:05.600018 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:05.599528    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m" event={"ID":"646879f7-5a4d-49c8-93e2-26fabf23905f","Type":"ContainerDied","Data":"9a782b791a76cd78fa9575872c565cce2a0ee7fe19642dfb9bb122c45c17a4f2"}
Oct 31 00:12:06.783534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.783504    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:06.845988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.845949    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\") pod \"646879f7-5a4d-49c8-93e2-26fabf23905f\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") "
Oct 31 00:12:06.846186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.846012    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\" (UniqueName: \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\") pod \"646879f7-5a4d-49c8-93e2-26fabf23905f\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") "
Oct 31 00:12:06.846186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.846128    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\") pod \"646879f7-5a4d-49c8-93e2-26fabf23905f\" (UID: \"646879f7-5a4d-49c8-93e2-26fabf23905f\") "
Oct 31 00:12:06.846526 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.846491    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca" pod "646879f7-5a4d-49c8-93e2-26fabf23905f" (UID: "646879f7-5a4d-49c8-93e2-26fabf23905f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:12:06.846804 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.846786    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:06.848803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.848776    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca" pod "646879f7-5a4d-49c8-93e2-26fabf23905f" (UID: "646879f7-5a4d-49c8-93e2-26fabf23905f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:06.848902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.848810    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v" pod "646879f7-5a4d-49c8-93e2-26fabf23905f" (UID: "646879f7-5a4d-49c8-93e2-26fabf23905f"). InnerVolumeSpecName "kube-api-access-fb25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:06.947983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.947938    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/646879f7-5a4d-49c8-93e2-26fabf23905f-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:06.947983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:06.947976    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb25v\" (UniqueName: \"kubernetes.io/projected/646879f7-5a4d-49c8-93e2-26fabf23905f-kube-api-access-fb25v\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:07.615211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:07.615181    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"
Oct 31 00:12:07.615211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:07.615190    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m" event={"ID":"646879f7-5a4d-49c8-93e2-26fabf23905f","Type":"ContainerDied","Data":"f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f"}
Oct 31 00:12:07.615476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:07.615242    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e2248e5e6a76dd925a85bd20ca92ec733f10e3c561d93c50342710fab4d38f"
Oct 31 00:12:16.972636 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:12:16.972593    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:12:21.310314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.310275    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"]
Oct 31 00:12:21.311066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.311012    6033 cpu_manager.go:414] "containerMap: removing container" podUID="646879f7-5a4d-49c8-93e2-26fabf23905f" containerName="revoke-keys"
Oct 31 00:12:21.311066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.311043    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646879f7-5a4d-49c8-93e2-26fabf23905f" containerName="revoke-keys"
Oct 31 00:12:21.311171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.311158    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="646879f7-5a4d-49c8-93e2-26fabf23905f" containerName="revoke-keys"
Oct 31 00:12:21.314439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.314411    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.364733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.364691    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:12:21.369150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.369114    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"]
Oct 31 00:12:21.378477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.378435    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.378680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.378633    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.378852 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.378833    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmr9\" (UniqueName: \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.413397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.413359    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:12:21.480728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.480685    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmr9\" (UniqueName: \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.480947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.480762    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.480947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.480932    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.481517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.481487    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.483375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.483355    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.539239 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.539209    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmr9\" (UniqueName: \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\") pod \"maas-api-revoke-sub-20e3177f-kdlnq\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:21.629097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:21.628962    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:22.008109 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:22.008083    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"]
Oct 31 00:12:22.010599 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:12:22.010558    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c35045_b469_45f7_8b20_b31991827768.slice/crio-638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af WatchSource:0}: Error finding container 638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af: Status 404 returned error can't find the container with id 638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af
Oct 31 00:12:22.737097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:22.736985    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq" event={"ID":"c7c35045-b469-45f7-8b20-b31991827768","Type":"ContainerStarted","Data":"e225faadfc0059d8ec5febf020f65699bba31111be49059145a00d5bd3f68211"}
Oct 31 00:12:22.737097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:22.737082    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq" event={"ID":"c7c35045-b469-45f7-8b20-b31991827768","Type":"ContainerStarted","Data":"638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af"}
Oct 31 00:12:22.847147 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:22.847080    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq" podStartSLOduration=1.847062471 podStartE2EDuration="1.847062471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:22.84609759 +0000 UTC m=+1795.193081488" watchObservedRunningTime="2026-10-31 00:12:22.847062471 +0000 UTC m=+1795.194046371"
Oct 31 00:12:23.745100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:23.745065    6033 generic.go:356] "Generic (PLEG): container finished" podID="c7c35045-b469-45f7-8b20-b31991827768" containerID="e225faadfc0059d8ec5febf020f65699bba31111be49059145a00d5bd3f68211" exitCode=0
Oct 31 00:12:23.745561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:23.745166    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq" event={"ID":"c7c35045-b469-45f7-8b20-b31991827768","Type":"ContainerDied","Data":"e225faadfc0059d8ec5febf020f65699bba31111be49059145a00d5bd3f68211"}
Oct 31 00:12:24.925441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:24.925404    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:25.033379 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.033264    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\" (UniqueName: \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\") pod \"c7c35045-b469-45f7-8b20-b31991827768\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") "
Oct 31 00:12:25.033379 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.033325    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\") pod \"c7c35045-b469-45f7-8b20-b31991827768\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") "
Oct 31 00:12:25.033379 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.033353    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\") pod \"c7c35045-b469-45f7-8b20-b31991827768\" (UID: \"c7c35045-b469-45f7-8b20-b31991827768\") "
Oct 31 00:12:25.033826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.033798    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca" pod "c7c35045-b469-45f7-8b20-b31991827768" (UID: "c7c35045-b469-45f7-8b20-b31991827768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:12:25.033996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.033979    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7c35045-b469-45f7-8b20-b31991827768-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:25.035658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.035629    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca" pod "c7c35045-b469-45f7-8b20-b31991827768" (UID: "c7c35045-b469-45f7-8b20-b31991827768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:25.035913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.035884    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9" pod "c7c35045-b469-45f7-8b20-b31991827768" (UID: "c7c35045-b469-45f7-8b20-b31991827768"). InnerVolumeSpecName "kube-api-access-xkmr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:25.135142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.135106    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmr9\" (UniqueName: \"kubernetes.io/projected/c7c35045-b469-45f7-8b20-b31991827768-kube-api-access-xkmr9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:25.135142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.135140    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7c35045-b469-45f7-8b20-b31991827768-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:12:25.760545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.760511    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"
Oct 31 00:12:25.760752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.760541    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq" event={"ID":"c7c35045-b469-45f7-8b20-b31991827768","Type":"ContainerDied","Data":"638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af"}
Oct 31 00:12:25.760752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:25.760613    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638aa429035e44136c22cc83b71503dd6a808cac605d15eec8abb884257088af"
Oct 31 00:12:27.749625 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:12:27.749584    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:12:46.973559 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:12:46.973462    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:13:09.554104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:09.554067    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:13:16.972089 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:13:16.972049    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:13:45.647070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.647010    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"]
Oct 31 00:13:45.652318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.651345    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c7c35045-b469-45f7-8b20-b31991827768" containerName="revoke-keys"
Oct 31 00:13:45.652318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.651378    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c35045-b469-45f7-8b20-b31991827768" containerName="revoke-keys"
Oct 31 00:13:45.652318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.651566    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c35045-b469-45f7-8b20-b31991827768" containerName="revoke-keys"
Oct 31 00:13:45.656416 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.656380    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.697169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.697001    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.697377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.697258    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.697500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.697480    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4hl\" (UniqueName: \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.705828 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.705795    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:13:45.715148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.715111    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"]
Oct 31 00:13:45.738042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.737992    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:13:45.816187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.815684    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.816670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.816644    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.855120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.848867    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.855120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.849079    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4hl\" (UniqueName: \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.899112 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.892762    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.927052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.918773    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4hl\" (UniqueName: \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\") pod \"maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:45.976102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:45.975144    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:46.429066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:46.428404    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"]
Oct 31 00:13:46.434723 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:13:46.434685    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab35163_5c3a_4863_b512_064fe64c5694.slice/crio-97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374 WatchSource:0}: Error finding container 97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374: Status 404 returned error can't find the container with id 97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374
Oct 31 00:13:46.973401 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:13:46.973365    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:13:47.348128 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:47.348089    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj" event={"ID":"4ab35163-5c3a-4863-b512-064fe64c5694","Type":"ContainerStarted","Data":"05f5338fce009191eb7d8a5d9d4125797c69fcdd8c55ea48856d5ac4425df871"}
Oct 31 00:13:47.348317 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:47.348148    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj" event={"ID":"4ab35163-5c3a-4863-b512-064fe64c5694","Type":"ContainerStarted","Data":"97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374"}
Oct 31 00:13:47.462937 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:47.462856    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj" podStartSLOduration=2.462833098 podStartE2EDuration="2.462833098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:47.456717918 +0000 UTC m=+1879.803701815" watchObservedRunningTime="2026-10-31 00:13:47.462833098 +0000 UTC m=+1879.809816997"
Oct 31 00:13:48.356900 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:48.356860    6033 generic.go:356] "Generic (PLEG): container finished" podID="4ab35163-5c3a-4863-b512-064fe64c5694" containerID="05f5338fce009191eb7d8a5d9d4125797c69fcdd8c55ea48856d5ac4425df871" exitCode=0
Oct 31 00:13:48.357366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:48.356922    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj" event={"ID":"4ab35163-5c3a-4863-b512-064fe64c5694","Type":"ContainerDied","Data":"05f5338fce009191eb7d8a5d9d4125797c69fcdd8c55ea48856d5ac4425df871"}
Oct 31 00:13:49.547734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.547704    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:49.714961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.714869    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\") pod \"4ab35163-5c3a-4863-b512-064fe64c5694\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") "
Oct 31 00:13:49.715188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.715048    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\" (UniqueName: \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\") pod \"4ab35163-5c3a-4863-b512-064fe64c5694\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") "
Oct 31 00:13:49.715188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.715173    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\") pod \"4ab35163-5c3a-4863-b512-064fe64c5694\" (UID: \"4ab35163-5c3a-4863-b512-064fe64c5694\") "
Oct 31 00:13:49.715383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.715352    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca" pod "4ab35163-5c3a-4863-b512-064fe64c5694" (UID: "4ab35163-5c3a-4863-b512-064fe64c5694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:13:49.715944 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.715918    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:49.717968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.717936    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca" pod "4ab35163-5c3a-4863-b512-064fe64c5694" (UID: "4ab35163-5c3a-4863-b512-064fe64c5694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:49.718144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.718016    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl" pod "4ab35163-5c3a-4863-b512-064fe64c5694" (UID: "4ab35163-5c3a-4863-b512-064fe64c5694"). InnerVolumeSpecName "kube-api-access-2j4hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:49.817642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.817589    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4hl\" (UniqueName: \"kubernetes.io/projected/4ab35163-5c3a-4863-b512-064fe64c5694-kube-api-access-2j4hl\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:49.817642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:49.817630    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab35163-5c3a-4863-b512-064fe64c5694-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:50.378991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:50.378962    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"
Oct 31 00:13:50.379221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:50.379000    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj" event={"ID":"4ab35163-5c3a-4863-b512-064fe64c5694","Type":"ContainerDied","Data":"97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374"}
Oct 31 00:13:50.379221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:50.379088    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97b85d7228307cc575feeb7efb39857675f1b308eac993d1680f1757a3e0e374"
Oct 31 00:13:51.967020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:51.966982    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:13:51.967508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:51.967284    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" podUID="ce44ee65-6516-4c15-9802-9e917636548d" containerName="maas-api" containerID="cri-o://b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293" gracePeriod=30
Oct 31 00:13:52.350772 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.350737    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:13:52.372402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.372364    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") pod \"ce44ee65-6516-4c15-9802-9e917636548d\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") "
Oct 31 00:13:52.374788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.374752    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls" pod "ce44ee65-6516-4c15-9802-9e917636548d" (UID: "ce44ee65-6516-4c15-9802-9e917636548d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:52.395651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.395617    6033 generic.go:356] "Generic (PLEG): container finished" podID="ce44ee65-6516-4c15-9802-9e917636548d" containerID="b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293" exitCode=0
Oct 31 00:13:52.395860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.395682    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"
Oct 31 00:13:52.395860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.395688    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" event={"ID":"ce44ee65-6516-4c15-9802-9e917636548d","Type":"ContainerDied","Data":"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"}
Oct 31 00:13:52.395860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.395805    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2" event={"ID":"ce44ee65-6516-4c15-9802-9e917636548d","Type":"ContainerDied","Data":"d94a1eb60e79c0e5757cf3d4d2fec9e936d2eec59f6318abe7095e7980bde4cd"}
Oct 31 00:13:52.395860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.395828    6033 scope.go:122] "RemoveContainer" containerID="b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"
Oct 31 00:13:52.407044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.407000    6033 scope.go:122] "RemoveContainer" containerID="b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"
Oct 31 00:13:52.407478 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:13:52.407444    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293\": container with ID starting with b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293 not found: ID does not exist" containerID="b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"
Oct 31 00:13:52.407591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.407490    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293"} err="failed to get container status \"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293\": rpc error: code = NotFound desc = could not find container \"b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293\": container with ID starting with b678fb8771574f6683e69f1e336d8a0eb58b48849f876e0d3da36710dea0f293 not found: ID does not exist"
Oct 31 00:13:52.475116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.474261    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\") pod \"ce44ee65-6516-4c15-9802-9e917636548d\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") "
Oct 31 00:13:52.475116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.474458    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\" (UniqueName: \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\") pod \"ce44ee65-6516-4c15-9802-9e917636548d\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") "
Oct 31 00:13:52.475116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.474685    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\") pod \"ce44ee65-6516-4c15-9802-9e917636548d\" (UID: \"ce44ee65-6516-4c15-9802-9e917636548d\") "
Oct 31 00:13:52.475116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.475025    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca" pod "ce44ee65-6516-4c15-9802-9e917636548d" (UID: "ce44ee65-6516-4c15-9802-9e917636548d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:13:52.475456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.475388    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:52.475456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.475412    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce44ee65-6516-4c15-9802-9e917636548d-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:52.501103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.497678    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs" pod "ce44ee65-6516-4c15-9802-9e917636548d" (UID: "ce44ee65-6516-4c15-9802-9e917636548d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:52.577443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.576892    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce44ee65-6516-4c15-9802-9e917636548d-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:52.590457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.577646    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9" pod "ce44ee65-6516-4c15-9802-9e917636548d" (UID: "ce44ee65-6516-4c15-9802-9e917636548d"). InnerVolumeSpecName "kube-api-access-9pcw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:52.678301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.678257    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcw9\" (UniqueName: \"kubernetes.io/projected/ce44ee65-6516-4c15-9802-9e917636548d-kube-api-access-9pcw9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:13:52.817538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.817493    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:13:52.820561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:52.820526    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-6bsh2"]
Oct 31 00:13:53.831875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:53.831828    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce44ee65-6516-4c15-9802-9e917636548d" path="/var/lib/kubelet/pods/ce44ee65-6516-4c15-9802-9e917636548d/volumes"
Oct 31 00:13:54.501416 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:54.501373    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"]
Oct 31 00:13:54.548366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:54.548334    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-gfl5m"]
Oct 31 00:13:55.831020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:55.830985    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646879f7-5a4d-49c8-93e2-26fabf23905f" path="/var/lib/kubelet/pods/646879f7-5a4d-49c8-93e2-26fabf23905f/volumes"
Oct 31 00:13:56.092928 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:13:56.092822    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:14:01.113054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.096966    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" podUID="d4a6369d-d2f2-4a63-ab41-e42116774e76" containerName="payload-processing" containerID="cri-o://05e2683f4df29d7f2ca123d888bfdd456d9e4b6f3d0ca9b78b2ba6d00c458f82" gracePeriod=25
Oct 31 00:14:01.525795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.525757    6033 generic.go:356] "Generic (PLEG): container finished" podID="d4a6369d-d2f2-4a63-ab41-e42116774e76" containerID="05e2683f4df29d7f2ca123d888bfdd456d9e4b6f3d0ca9b78b2ba6d00c458f82" exitCode=0
Oct 31 00:14:01.526010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.525833    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" event={"ID":"d4a6369d-d2f2-4a63-ab41-e42116774e76","Type":"ContainerDied","Data":"05e2683f4df29d7f2ca123d888bfdd456d9e4b6f3d0ca9b78b2ba6d00c458f82"}
Oct 31 00:14:01.622021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.621983    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:14:01.681672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.681637    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\") pod \"d4a6369d-d2f2-4a63-ab41-e42116774e76\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") "
Oct 31 00:14:01.681875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.681857    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\" (UniqueName: \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\") pod \"d4a6369d-d2f2-4a63-ab41-e42116774e76\" (UID: \"d4a6369d-d2f2-4a63-ab41-e42116774e76\") "
Oct 31 00:14:01.682073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.682049    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume" pod "d4a6369d-d2f2-4a63-ab41-e42116774e76" (UID: "d4a6369d-d2f2-4a63-ab41-e42116774e76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:01.682385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.682353    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4a6369d-d2f2-4a63-ab41-e42116774e76-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:01.684245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.684210    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn" pod "d4a6369d-d2f2-4a63-ab41-e42116774e76" (UID: "d4a6369d-d2f2-4a63-ab41-e42116774e76"). InnerVolumeSpecName "kube-api-access-fqfbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:01.784487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:01.784444    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfbn\" (UniqueName: \"kubernetes.io/projected/d4a6369d-d2f2-4a63-ab41-e42116774e76-kube-api-access-fqfbn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:02.534819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:02.534790    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"
Oct 31 00:14:02.535369 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:02.534781    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh" event={"ID":"d4a6369d-d2f2-4a63-ab41-e42116774e76","Type":"ContainerDied","Data":"3ea46bf5dfb7abe4ac71a0ac0ab231bc0cdf839ecc444c0b961cecf29b6314a6"}
Oct 31 00:14:02.535369 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:02.534940    6033 scope.go:122] "RemoveContainer" containerID="05e2683f4df29d7f2ca123d888bfdd456d9e4b6f3d0ca9b78b2ba6d00c458f82"
Oct 31 00:14:02.693044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:02.692993    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:14:02.738985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:02.738940    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71b76a-7b898f6878-hmdwh"]
Oct 31 00:14:03.831532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:03.831496    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a6369d-d2f2-4a63-ab41-e42116774e76" path="/var/lib/kubelet/pods/d4a6369d-d2f2-4a63-ab41-e42116774e76/volumes"
Oct 31 00:14:06.863786 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.863690    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:14:06.864812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864704    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d4a6369d-d2f2-4a63-ab41-e42116774e76" containerName="payload-processing"
Oct 31 00:14:06.864812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864745    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a6369d-d2f2-4a63-ab41-e42116774e76" containerName="payload-processing"
Oct 31 00:14:06.864812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864772    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ce44ee65-6516-4c15-9802-9e917636548d" containerName="maas-api"
Oct 31 00:14:06.864812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864781    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce44ee65-6516-4c15-9802-9e917636548d" containerName="maas-api"
Oct 31 00:14:06.865323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864818    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4ab35163-5c3a-4863-b512-064fe64c5694" containerName="revoke-keys"
Oct 31 00:14:06.865323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.864828    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab35163-5c3a-4863-b512-064fe64c5694" containerName="revoke-keys"
Oct 31 00:14:06.865323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.865045    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab35163-5c3a-4863-b512-064fe64c5694" containerName="revoke-keys"
Oct 31 00:14:06.865323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.865077    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a6369d-d2f2-4a63-ab41-e42116774e76" containerName="payload-processing"
Oct 31 00:14:06.865323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.865092    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce44ee65-6516-4c15-9802-9e917636548d" containerName="maas-api"
Oct 31 00:14:06.872007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.871871    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:06.922819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.922776    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:14:06.963672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.963633    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a251fdfb\""
Oct 31 00:14:06.963930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.963823    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a251fdfb-dockercfg-nmpq4\""
Oct 31 00:14:06.988171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.988127    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nd8\" (UniqueName: \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:06.988397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:06.988231    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.095080 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.091261    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nd8\" (UniqueName: \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.095080 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.091371    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.095080 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.092447    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.152216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.151179    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nd8\" (UniqueName: \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\") pod \"payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.194516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.194475    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:07.591770 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.591735    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:14:07.592235 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:14:07.592203    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd217a9_2414_4b86_bc25_f8462678e92e.slice/crio-3ee2bc24aa4f692cb825c2866e33ab1b51f7bb530199ee06df34da2cc2e24b27 WatchSource:0}: Error finding container 3ee2bc24aa4f692cb825c2866e33ab1b51f7bb530199ee06df34da2cc2e24b27: Status 404 returned error can't find the container with id 3ee2bc24aa4f692cb825c2866e33ab1b51f7bb530199ee06df34da2cc2e24b27
Oct 31 00:14:07.621062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:07.621006    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" event={"ID":"dfd217a9-2414-4b86-bc25-f8462678e92e","Type":"ContainerStarted","Data":"3ee2bc24aa4f692cb825c2866e33ab1b51f7bb530199ee06df34da2cc2e24b27"}
Oct 31 00:14:08.630573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:08.630534    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" event={"ID":"dfd217a9-2414-4b86-bc25-f8462678e92e","Type":"ContainerStarted","Data":"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"}
Oct 31 00:14:08.631016 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:08.630621    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:08.750953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:08.750894    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" podStartSLOduration=2.750872862 podStartE2EDuration="2.750872862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:08.749699031 +0000 UTC m=+1901.096682926" watchObservedRunningTime="2026-10-31 00:14:08.750872862 +0000 UTC m=+1901.097856761"
Oct 31 00:14:09.139677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:09.139633    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:14:13.739242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.739204    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:14:13.747091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.745997    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:13.795745 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.795703    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d9e782-serving-cert\""
Oct 31 00:14:13.851848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.851812    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:14:13.919251 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.919209    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:13.919481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.919454    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:13.919538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.919530    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:13.919614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:13.919595    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcfg\" (UniqueName: \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.021196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.021155    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.021196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.021208    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcfg\" (UniqueName: \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.024283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.024252    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.029201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.029162    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.029495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.029471    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.029667 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:14.029641    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-serving-cert: secret "maas-api-e2e-shared-b-w3-d9e782-serving-cert" not found
Oct 31 00:14:14.029768 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:14.029752    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls podName:5b1d2b89-0c87-4201-a564-d3a02da3a5d6 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:14.529725431 +0000 UTC m=+1906.876709322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" (UID: "5b1d2b89-0c87-4201-a564-d3a02da3a5d6") : secret "maas-api-e2e-shared-b-w3-d9e782-serving-cert" not found
Oct 31 00:14:14.029908 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.029884    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.079847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.079810    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcfg\" (UniqueName: \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.538685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.538631    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.541475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.541438    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:14.664127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:14.664088    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:15.046925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:15.046882    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:14:15.047757 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:14:15.047715    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1d2b89_0c87_4201_a564_d3a02da3a5d6.slice/crio-92d25a2a5abd16ec0f30063c1b70ad93ae09e0b8655c080cb70bd4aa1ebe4812 WatchSource:0}: Error finding container 92d25a2a5abd16ec0f30063c1b70ad93ae09e0b8655c080cb70bd4aa1ebe4812: Status 404 returned error can't find the container with id 92d25a2a5abd16ec0f30063c1b70ad93ae09e0b8655c080cb70bd4aa1ebe4812
Oct 31 00:14:15.769345 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:15.769301    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" event={"ID":"5b1d2b89-0c87-4201-a564-d3a02da3a5d6","Type":"ContainerStarted","Data":"92d25a2a5abd16ec0f30063c1b70ad93ae09e0b8655c080cb70bd4aa1ebe4812"}
Oct 31 00:14:16.804844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:16.804793    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" event={"ID":"5b1d2b89-0c87-4201-a564-d3a02da3a5d6","Type":"ContainerStarted","Data":"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"}
Oct 31 00:14:16.805531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:16.804913    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:16.916877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:16.916813    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" podStartSLOduration=2.940001075 podStartE2EDuration="3.916791062s" totalImagesPullingTime="976.789987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:16.915757378 +0000 UTC m=+1909.262741279" watchObservedRunningTime="2026-10-31 00:14:16.916791062 +0000 UTC m=+1909.263774961"
Oct 31 00:14:16.973137 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:16.973097    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:14:19.641763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:19.641726    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:14:22.818929 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:22.818896    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:14:30.186998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:30.186958    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 31 00:14:30.187709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:30.187268    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy" containerID="cri-o://4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e" gracePeriod=30
Oct 31 00:14:34.975115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:34.975078    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 31 00:14:34.975624 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:34.975334    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy" containerID="cri-o://f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab" gracePeriod=30
Oct 31 00:14:35.163791 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.163751    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 31 00:14:35.164022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.163993    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" containerID="cri-o://f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87" gracePeriod=30
Oct 31 00:14:35.491771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.491706    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 31 00:14:35.593650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593609    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.593834 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593730    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.593834 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593765    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.593834 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593820    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.593961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593869    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.593961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.593898    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.594110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594092    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.594180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594126    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594242    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594284    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594301    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594364    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\") pod \"7ea30a07-c3e3-45c0-8f8f-36b242237879\" (UID: \"7ea30a07-c3e3-45c0-8f8f-36b242237879\") "
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594439    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:35.594635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.594595    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.595182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.595157    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.595264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.595185    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.595264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.595201    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.595264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.595213    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea30a07-c3e3-45c0-8f8f-36b242237879-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.595264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.595226    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.596490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.596464    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:35.596592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.596509    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "kube-api-access-k4mwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:35.596704 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.596678    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:35.596757 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.596732    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token" pod "7ea30a07-c3e3-45c0-8f8f-36b242237879" (UID: "7ea30a07-c3e3-45c0-8f8f-36b242237879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:35.696336 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.696299    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.696336 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.696336    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4mwm\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-kube-api-access-k4mwm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.696549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.696353    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.696549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.696366    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea30a07-c3e3-45c0-8f8f-36b242237879-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:35.901904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.901862    6033 patch_prober.go:29] interesting pod/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:14:35.902091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.901937    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:14:35.950768 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.950732    6033 generic.go:356] "Generic (PLEG): container finished" podID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerID="4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e" exitCode=0
Oct 31 00:14:35.950939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.950780    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"
Oct 31 00:14:35.950939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.950815    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" event={"ID":"7ea30a07-c3e3-45c0-8f8f-36b242237879","Type":"ContainerDied","Data":"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"}
Oct 31 00:14:35.950939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.950878    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt" event={"ID":"7ea30a07-c3e3-45c0-8f8f-36b242237879","Type":"ContainerDied","Data":"939e345c0c7a0334b652d510f3e61e04b000f47c66145ad94015016a4db1c7ff"}
Oct 31 00:14:35.950939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.950908    6033 scope.go:122] "RemoveContainer" containerID="4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"
Oct 31 00:14:35.962743 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.962722    6033 scope.go:122] "RemoveContainer" containerID="4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"
Oct 31 00:14:35.963102 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:35.963073    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e\": container with ID starting with 4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e not found: ID does not exist" containerID="4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"
Oct 31 00:14:35.963200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:35.963119    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e"} err="failed to get container status \"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e\": rpc error: code = NotFound desc = could not find container \"4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e\": container with ID starting with 4a9cd98a9bb90eece62b74c6d71bfeab8fc8e7d9bd1033bc449ffd98a6898c9e not found: ID does not exist"
Oct 31 00:14:36.079113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.079065    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"]
Oct 31 00:14:36.080081 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.080063    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy"
Oct 31 00:14:36.080155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.080083    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy"
Oct 31 00:14:36.080300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.080289    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" containerName="istio-proxy"
Oct 31 00:14:36.086007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.085980    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.110955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.110911    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 31 00:14:36.124124 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.124085    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-hnqpt"]
Oct 31 00:14:36.154072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.154020    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:14:36.155233 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.155206    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:14:36.157673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.157645    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"]
Oct 31 00:14:36.203487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.203451    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gsr\" (UniqueName: \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.203714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.203619    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.203806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.203790    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.243321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.243288    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:14:36.243527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.243503    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" containerID="cri-o://b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab" gracePeriod=30
Oct 31 00:14:36.305210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.305170    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.305394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.305263    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gsr\" (UniqueName: \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.305394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.305361    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.305957 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.305926    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.307803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.307778    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.375463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.375370    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gsr\" (UniqueName: \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.400932 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.400893    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:36.777257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.777224    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"]
Oct 31 00:14:36.782832 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:14:36.782750    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b13a87_fa22_4f30_a443_c893e9bf5f8e.slice/crio-401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3 WatchSource:0}: Error finding container 401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3: Status 404 returned error can't find the container with id 401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3
Oct 31 00:14:36.960161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:36.960125    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d" event={"ID":"33b13a87-fa22-4f30-a443-c893e9bf5f8e","Type":"ContainerStarted","Data":"401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3"}
Oct 31 00:14:37.832250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:37.832220    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea30a07-c3e3-45c0-8f8f-36b242237879" path="/var/lib/kubelet/pods/7ea30a07-c3e3-45c0-8f8f-36b242237879/volumes"
Oct 31 00:14:37.971251 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:37.971211    6033 generic.go:356] "Generic (PLEG): container finished" podID="33b13a87-fa22-4f30-a443-c893e9bf5f8e" containerID="e1a73c874148699c5a9eb55256775e2c1f4bf40ae95e3f96b14b503cc4c61f22" exitCode=0
Oct 31 00:14:37.971448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:37.971369    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d" event={"ID":"33b13a87-fa22-4f30-a443-c893e9bf5f8e","Type":"ContainerDied","Data":"e1a73c874148699c5a9eb55256775e2c1f4bf40ae95e3f96b14b503cc4c61f22"}
Oct 31 00:14:39.058193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.058151    6033 patch_prober.go:29] interesting pod/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:14:39.058683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.058247    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:14:39.161608 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.161575    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:39.262570 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.262527    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\") pod \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") "
Oct 31 00:14:39.262769 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.262704    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\") pod \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") "
Oct 31 00:14:39.262869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.262851    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\" (UniqueName: \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\") pod \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\" (UID: \"33b13a87-fa22-4f30-a443-c893e9bf5f8e\") "
Oct 31 00:14:39.263171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.263134    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca" pod "33b13a87-fa22-4f30-a443-c893e9bf5f8e" (UID: "33b13a87-fa22-4f30-a443-c893e9bf5f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:39.263539 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.263515    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:39.264854 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.264831    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca" pod "33b13a87-fa22-4f30-a443-c893e9bf5f8e" (UID: "33b13a87-fa22-4f30-a443-c893e9bf5f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:14:39.264952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.264932    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr" pod "33b13a87-fa22-4f30-a443-c893e9bf5f8e" (UID: "33b13a87-fa22-4f30-a443-c893e9bf5f8e"). InnerVolumeSpecName "kube-api-access-p2gsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:39.365132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.365088    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gsr\" (UniqueName: \"kubernetes.io/projected/33b13a87-fa22-4f30-a443-c893e9bf5f8e-kube-api-access-p2gsr\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:39.365132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.365131    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b13a87-fa22-4f30-a443-c893e9bf5f8e-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:39.988454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.987708    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d" event={"ID":"33b13a87-fa22-4f30-a443-c893e9bf5f8e","Type":"ContainerDied","Data":"401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3"}
Oct 31 00:14:39.988454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.987763    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"
Oct 31 00:14:39.988454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:39.987773    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401a008ac73b161205016f29698f8ee977f061d250795445b8e09e10216114a3"
Oct 31 00:14:40.019710 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:40.019679    6033 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558c7410_471f_42f3_b60a_e214a76117ee.slice/crio-f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 00:14:40.302072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.302023    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 31 00:14:40.383601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383565    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.383796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383657    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.383796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383726    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.383796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383769    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.383967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383812    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.383967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383957    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.384134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.383959    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.384134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384008    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.384134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384061    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.384134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384134    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\") pod \"558c7410-471f-42f3-b60a-e214a76117ee\" (UID: \"558c7410-471f-42f3-b60a-e214a76117ee\") "
Oct 31 00:14:40.384441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384225    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.384863 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384835    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.384975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.384865    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.385568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.385494    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.385568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.385542    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.385868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.385832    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:40.388492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.388468    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.388630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.388548    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:40.392510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.392464    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "kube-api-access-6w5h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:40.394865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.394838    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo" pod "558c7410-471f-42f3-b60a-e214a76117ee" (UID: "558c7410-471f-42f3-b60a-e214a76117ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:40.443682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.443652    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 31 00:14:40.486685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486645    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558c7410-471f-42f3-b60a-e214a76117ee-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486689    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558c7410-471f-42f3-b60a-e214a76117ee-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486705    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486719    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w5h9\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-kube-api-access-6w5h9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486735    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486749    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558c7410-471f-42f3-b60a-e214a76117ee-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.486887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.486763    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558c7410-471f-42f3-b60a-e214a76117ee-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.587544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587495    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587564    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587720    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587789    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587838    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587887    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.587921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587896    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.588133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.587932    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.588133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.588020    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.588133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.588083    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\") pod \"30985f81-128c-40db-8c21-68d9fb835c35\" (UID: \"30985f81-128c-40db-8c21-68d9fb835c35\") "
Oct 31 00:14:40.588367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.588343    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.588962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.588938    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.589095 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.588965    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.589341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.589319    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:40.589594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.589572    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.590075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.590018    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.590664 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.590639    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:40.590895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.590861    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:40.591005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.590982    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:40.591895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.591866    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz" pod "30985f81-128c-40db-8c21-68d9fb835c35" (UID: "30985f81-128c-40db-8c21-68d9fb835c35"). InnerVolumeSpecName "kube-api-access-k85dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690631    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690666    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30985f81-128c-40db-8c21-68d9fb835c35-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690676    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85dz\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-kube-api-access-k85dz\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690688    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690697    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30985f81-128c-40db-8c21-68d9fb835c35-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690705    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30985f81-128c-40db-8c21-68d9fb835c35-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.690722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.690713    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30985f81-128c-40db-8c21-68d9fb835c35-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:40.996175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.996079    6033 generic.go:356] "Generic (PLEG): container finished" podID="558c7410-471f-42f3-b60a-e214a76117ee" containerID="f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab" exitCode=0
Oct 31 00:14:40.996175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.996121    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"
Oct 31 00:14:40.996175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.996157    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" event={"ID":"558c7410-471f-42f3-b60a-e214a76117ee","Type":"ContainerDied","Data":"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"}
Oct 31 00:14:40.996458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.996214    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw" event={"ID":"558c7410-471f-42f3-b60a-e214a76117ee","Type":"ContainerDied","Data":"6b5b1501755402a102473290a2b88c0c29de55842bf333db84144724eb57d352"}
Oct 31 00:14:40.996458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.996244    6033 scope.go:122] "RemoveContainer" containerID="f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"
Oct 31 00:14:40.997861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.997833    6033 generic.go:356] "Generic (PLEG): container finished" podID="30985f81-128c-40db-8c21-68d9fb835c35" containerID="f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87" exitCode=0
Oct 31 00:14:40.997975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.997884    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"
Oct 31 00:14:40.997975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.997917    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" event={"ID":"30985f81-128c-40db-8c21-68d9fb835c35","Type":"ContainerDied","Data":"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"}
Oct 31 00:14:40.997975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:40.997967    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4" event={"ID":"30985f81-128c-40db-8c21-68d9fb835c35","Type":"ContainerDied","Data":"18447e3abe385eb6797f38ecc19b56c9bd6f50248f2b6c9fc8bbbc7a2389312e"}
Oct 31 00:14:41.009567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.009542    6033 scope.go:122] "RemoveContainer" containerID="f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"
Oct 31 00:14:41.009880 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:41.009861    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab\": container with ID starting with f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab not found: ID does not exist" containerID="f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"
Oct 31 00:14:41.009952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.009887    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab"} err="failed to get container status \"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab\": rpc error: code = NotFound desc = could not find container \"f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab\": container with ID starting with f88a3ac1fbe64bdcf7c440aa21af894abcf103e86e0c81e3136f79f73165dbab not found: ID does not exist"
Oct 31 00:14:41.009952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.009904    6033 scope.go:122] "RemoveContainer" containerID="f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"
Oct 31 00:14:41.020024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.020004    6033 scope.go:122] "RemoveContainer" containerID="f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"
Oct 31 00:14:41.020349 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:41.020329    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87\": container with ID starting with f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87 not found: ID does not exist" containerID="f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"
Oct 31 00:14:41.020398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.020361    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87"} err="failed to get container status \"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87\": rpc error: code = NotFound desc = could not find container \"f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87\": container with ID starting with f8f159fe8e32ae0b42651b6ca10fe3c599fcc90c9fdff9520d19e80e95a14a87 not found: ID does not exist"
Oct 31 00:14:41.153543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.153487    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 31 00:14:41.201912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.201872    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-mgtkw"]
Oct 31 00:14:41.265265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.265226    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 31 00:14:41.271895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.271859    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-z6jt4"]
Oct 31 00:14:41.539309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.539278    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:14:41.602750 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.602717    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.602930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.602797    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.602930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.602868    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.602973    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603060 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603013    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603073    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603278 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603256    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603306    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603346    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\") pod \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\" (UID: \"46dc8d0d-3428-4e39-9b48-61ab8cafb419\") "
Oct 31 00:14:41.603490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603369    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.603490 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603409    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.603597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603567    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.603597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603584    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.603851 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603827    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:41.603967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603951    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.604042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603973    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.604042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.603988    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.604042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.604004    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.604042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.604017    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.605545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.605520    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "kube-api-access-d52lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.605882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.605574    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:14:41.605882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.605650    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:14:41.605882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.605806    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token" pod "46dc8d0d-3428-4e39-9b48-61ab8cafb419" (UID: "46dc8d0d-3428-4e39-9b48-61ab8cafb419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.705546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.705498    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.705546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.705542    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.705546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.705551    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.705546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.705560    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52lr\" (UniqueName: \"kubernetes.io/projected/46dc8d0d-3428-4e39-9b48-61ab8cafb419-kube-api-access-d52lr\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.831287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.831202    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30985f81-128c-40db-8c21-68d9fb835c35" path="/var/lib/kubelet/pods/30985f81-128c-40db-8c21-68d9fb835c35/volumes"
Oct 31 00:14:41.831749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:41.831735    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558c7410-471f-42f3-b60a-e214a76117ee" path="/var/lib/kubelet/pods/558c7410-471f-42f3-b60a-e214a76117ee/volumes"
Oct 31 00:14:42.008468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.008435    6033 generic.go:356] "Generic (PLEG): container finished" podID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerID="b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab" exitCode=0
Oct 31 00:14:42.008683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.008489    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"
Oct 31 00:14:42.008683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.008525    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" event={"ID":"46dc8d0d-3428-4e39-9b48-61ab8cafb419","Type":"ContainerDied","Data":"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"}
Oct 31 00:14:42.008683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.008597    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz" event={"ID":"46dc8d0d-3428-4e39-9b48-61ab8cafb419","Type":"ContainerDied","Data":"18705d2f44680c7188c8da024cdeb044591fd16521d19ebca158df024e141249"}
Oct 31 00:14:42.008683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.008634    6033 scope.go:122] "RemoveContainer" containerID="b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"
Oct 31 00:14:42.019918 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.019896    6033 scope.go:122] "RemoveContainer" containerID="b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"
Oct 31 00:14:42.020355 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:42.020333    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab\": container with ID starting with b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab not found: ID does not exist" containerID="b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"
Oct 31 00:14:42.020446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.020368    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab"} err="failed to get container status \"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab\": rpc error: code = NotFound desc = could not find container \"b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab\": container with ID starting with b90b009c27956607bfc8527a09fa5c860798eb19aa54bc0868bf67d0c8a32bab not found: ID does not exist"
Oct 31 00:14:42.170650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.170559    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:14:42.217526 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:42.217477    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8qtdz"]
Oct 31 00:14:43.831653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:43.831620    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" path="/var/lib/kubelet/pods/46dc8d0d-3428-4e39-9b48-61ab8cafb419/volumes"
Oct 31 00:14:46.971899 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:14:46.971854    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:14:57.998469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:14:57.998419    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:15:00.161371 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.161331    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:00.162070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162053    6033 cpu_manager.go:414] "containerMap: removing container" podUID="33b13a87-fa22-4f30-a443-c893e9bf5f8e" containerName="revoke-keys"
Oct 31 00:15:00.162070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162071    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b13a87-fa22-4f30-a443-c893e9bf5f8e" containerName="revoke-keys"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162083    6033 cpu_manager.go:414] "containerMap: removing container" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162088    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162108    6033 cpu_manager.go:414] "containerMap: removing container" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162114    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162143    6033 cpu_manager.go:414] "containerMap: removing container" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy"
Oct 31 00:15:00.162161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162148    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy"
Oct 31 00:15:00.162395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162261    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b13a87-fa22-4f30-a443-c893e9bf5f8e" containerName="revoke-keys"
Oct 31 00:15:00.162395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162268    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="558c7410-471f-42f3-b60a-e214a76117ee" containerName="istio-proxy"
Oct 31 00:15:00.162395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162276    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="46dc8d0d-3428-4e39-9b48-61ab8cafb419" containerName="istio-proxy"
Oct 31 00:15:00.162395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.162286    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="30985f81-128c-40db-8c21-68d9fb835c35" containerName="istio-proxy"
Oct 31 00:15:00.166090 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.166025    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.178753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.178703    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:00.183764 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.183736    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:00.183942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.183847    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.188293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.188266    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.208402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.208368    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:00.213093 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.213065    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:00.213253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.213147    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.217638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.217614    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.224453 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.224425    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:15:00.225254 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.225224    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:00.230751 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.230729    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:00.230850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.230836    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.235293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.235271    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.255684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.255656    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:00.259993 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.259969    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.263562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.263539    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:00.268139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.268113    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.281039 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.280999    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:00.287540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.287507    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:00.290642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.290615    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:00.292423 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292392    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.292546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292447    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.292599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292580    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.292644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292631    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xl4\" (UniqueName: \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.292742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292713    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvbd\" (UniqueName: \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.292808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292788    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.292849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292826    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khljz\" (UniqueName: \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.292903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292848    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.292903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292871    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hp7h\" (UniqueName: \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.293003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.292943    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.293092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.293021    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8t6\" (UniqueName: \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.293214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.293195    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhsh\" (UniqueName: \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.293283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.293256    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.293374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.293354    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk22\" (UniqueName: \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.300921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.300889    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:00.305968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.305940    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.329252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.329219    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:00.368269 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.368207    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:00.374771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.373718    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:00.380567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.379820    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.394349    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396358    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvbd\" (UniqueName: \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396404    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396431    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khljz\" (UniqueName: \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396448    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396468    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hp7h\" (UniqueName: \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396491    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396593    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8t6\" (UniqueName: \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396625    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdn9\" (UniqueName: \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396670    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6kf\" (UniqueName: \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396789    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396833    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhsh\" (UniqueName: \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396887    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.396963    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397054    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk22\" (UniqueName: \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397082    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8j7p\" (UniqueName: \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397116    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397151    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397200    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397292    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.397579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397352    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xl4\" (UniqueName: \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.398733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.397729    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.398733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.398353    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.398733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.398394    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.399159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.399129    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.399352 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.399291    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.399961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.399927    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.400136 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.400112    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.428323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.426519    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:00.444703 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.444663    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:00.468848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.468810    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvbd\" (UniqueName: \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.469193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.469160    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khljz\" (UniqueName: \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.469521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.469462    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xl4\" (UniqueName: \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\") pod \"maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.472725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.470812    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hp7h\" (UniqueName: \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.472725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.470861    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk22\" (UniqueName: \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\") pod \"maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.472725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.471888    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhsh\" (UniqueName: \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.473672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.473642    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8t6\" (UniqueName: \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.481752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.481721    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:00.498581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498548    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdn9\" (UniqueName: \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.498709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498604    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6kf\" (UniqueName: \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.498709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498685    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.498821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498788    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.498886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498832    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8j7p\" (UniqueName: \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.498886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.498866    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.499440 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.499409    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.499580 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.499557    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.499642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.499586    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.501086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.501068    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:00.509237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.509212    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:00.529747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.529712    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:00.537549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.537513    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:00.558393 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.551729    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:00.567756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.560057    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:00.575300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.574595    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdn9\" (UniqueName: \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\") pod \"maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.577974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.576766    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8j7p\" (UniqueName: \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\") pod \"maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.588630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.585438    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:00.588630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.585906    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:00.594676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.590859    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6kf\" (UniqueName: \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\") pod \"maas-api-key-cleanup-29890095-fh9pl\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.635680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.635400    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:00.875995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:00.875961    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:00.879560 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:00.879528    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8ef086_21f1_4eb8_88d0_0bb7f5a60db4.slice/crio-e0d3e67afeccf0a46a2561867d801e125293cca838f59f1a0d004843cc29b425 WatchSource:0}: Error finding container e0d3e67afeccf0a46a2561867d801e125293cca838f59f1a0d004843cc29b425: Status 404 returned error can't find the container with id e0d3e67afeccf0a46a2561867d801e125293cca838f59f1a0d004843cc29b425
Oct 31 00:15:01.125444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.125412    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:01.128466 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:01.128437    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6819373_9571_4d18_9412_951f6f930c13.slice/crio-e962795a7fb7538565e0331624660193dfcadd881c222dca92fe30ef975195c3 WatchSource:0}: Error finding container e962795a7fb7538565e0331624660193dfcadd881c222dca92fe30ef975195c3: Status 404 returned error can't find the container with id e962795a7fb7538565e0331624660193dfcadd881c222dca92fe30ef975195c3
Oct 31 00:15:01.131483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.131455    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:01.131785 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:01.131762    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bfddbb_4c60_44a7_a122_4601fce98792.slice/crio-2fc7fb6fe952e1b1f9959d89239a9da91a27ca4af9333d127a8a37c817cbb61f WatchSource:0}: Error finding container 2fc7fb6fe952e1b1f9959d89239a9da91a27ca4af9333d127a8a37c817cbb61f: Status 404 returned error can't find the container with id 2fc7fb6fe952e1b1f9959d89239a9da91a27ca4af9333d127a8a37c817cbb61f
Oct 31 00:15:01.163944 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.163912    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerStarted","Data":"e962795a7fb7538565e0331624660193dfcadd881c222dca92fe30ef975195c3"}
Oct 31 00:15:01.165215 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.165193    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerStarted","Data":"2fc7fb6fe952e1b1f9959d89239a9da91a27ca4af9333d127a8a37c817cbb61f"}
Oct 31 00:15:01.166617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.166594    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerStarted","Data":"e0d3e67afeccf0a46a2561867d801e125293cca838f59f1a0d004843cc29b425"}
Oct 31 00:15:01.176655 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.176625    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:01.178381 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:01.178354    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bec1afd_a211_466b_9335_baaf35dd97a3.slice/crio-544ab11ac15160bc754eee1cf73fff0977f41ae9bf86b945720204ea94c1c9fc WatchSource:0}: Error finding container 544ab11ac15160bc754eee1cf73fff0977f41ae9bf86b945720204ea94c1c9fc: Status 404 returned error can't find the container with id 544ab11ac15160bc754eee1cf73fff0977f41ae9bf86b945720204ea94c1c9fc
Oct 31 00:15:01.188758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.188725    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:01.237953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.237918    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:01.257343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.257310    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:01.260501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.260473    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:01.306424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.306398    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:01.355768 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:01.355283    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:01.357356 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:01.357311    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14c80c7_70d7_4878_94c2_de9d7c7dd63e.slice/crio-d2fd25c30277edd204293b43edf93145af616fdc7ffbc25d1ed6d55dac511887 WatchSource:0}: Error finding container d2fd25c30277edd204293b43edf93145af616fdc7ffbc25d1ed6d55dac511887: Status 404 returned error can't find the container with id d2fd25c30277edd204293b43edf93145af616fdc7ffbc25d1ed6d55dac511887
Oct 31 00:15:02.176507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.176467    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerStarted","Data":"3cdd6e27a8bd5fff34b759d1ae7a56c7d69886c2f351f90539568f8bd1cdf911"}
Oct 31 00:15:02.177017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.176531    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerStarted","Data":"d2fd25c30277edd204293b43edf93145af616fdc7ffbc25d1ed6d55dac511887"}
Oct 31 00:15:02.178203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.178170    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerStarted","Data":"c41ad3e5d9c0d405ed3c795e61960299f3a644c7c680743cab7af3afdb3244d5"}
Oct 31 00:15:02.178334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.178226    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerStarted","Data":"bb2c290869bd9a03b72a7405abae896244daa102439d61f0237128ccae3f7c50"}
Oct 31 00:15:02.179832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.179805    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerStarted","Data":"fd856531a97a275e478bfd01462828e22401e649bade502603a61ef5f4734884"}
Oct 31 00:15:02.185271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.185241    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerStarted","Data":"bfb694d94325d1613a90083efff796c7aea9c87aa0a397f05b95c712a170dd94"}
Oct 31 00:15:02.188043 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.188003    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerStarted","Data":"ae06616c6d87a566a47c0d7ba21c03a93d58edc596b18e93baf0e85225c6040f"}
Oct 31 00:15:02.188166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.188093    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerStarted","Data":"19fa2102d6d39cbb2b59321cb6ee14d76666109ce2ad1b7af84ac313e52b3daf"}
Oct 31 00:15:02.190074 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.190013    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerStarted","Data":"366f7fa4d4ab9e2b4cf0b52dd2e677bc1dae56bba3d65f5e8c32a5444888c80f"}
Oct 31 00:15:02.190412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.190099    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerStarted","Data":"aa948befa7d7271ffb3f865c5542dcc615eeaae98a6b011df1e2834d7fdb0724"}
Oct 31 00:15:02.191942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.191917    6033 generic.go:356] "Generic (PLEG): container finished" podID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerID="e0fe8709b2309220e0257b2183045a4255f00f9cfa1fa6abc68d172ae9ded728" exitCode=60
Oct 31 00:15:02.192021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.191952    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerDied","Data":"e0fe8709b2309220e0257b2183045a4255f00f9cfa1fa6abc68d172ae9ded728"}
Oct 31 00:15:02.192301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.192280    6033 scope.go:122] "RemoveContainer" containerID="e0fe8709b2309220e0257b2183045a4255f00f9cfa1fa6abc68d172ae9ded728"
Oct 31 00:15:02.193736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.193714    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerStarted","Data":"7be47d5357a05afce559ec44add1c6bfe08962e0dae9c2a1fd51bd8346b8888c"}
Oct 31 00:15:02.193978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.193945    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerStarted","Data":"544ab11ac15160bc754eee1cf73fff0977f41ae9bf86b945720204ea94c1c9fc"}
Oct 31 00:15:02.195427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.195404    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerStarted","Data":"b368c212c9e613e42818094d86c18cebf8420aba2c7127fbfbd4c05cd7b44e57"}
Oct 31 00:15:02.195545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.195446    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerStarted","Data":"8165522a9a8542d1201504a30578212e9fff40b537ac2d3267509ad083222058"}
Oct 31 00:15:02.197123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.197103    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerStarted","Data":"6f908a75c28047114ca76f5239911263d44b1cc7125debc74ba25ca47a4090dd"}
Oct 31 00:15:02.197219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.197145    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerStarted","Data":"9756669b0fb5308161cc5bd9a266dccd698161c9a6d7053206f0df422aec4faa"}
Oct 31 00:15:02.291193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.290972    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" podStartSLOduration=2.290951444 podStartE2EDuration="2.290951444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.286024615 +0000 UTC m=+1954.633008515" watchObservedRunningTime="2026-10-31 00:15:02.290951444 +0000 UTC m=+1954.637935394"
Oct 31 00:15:02.400855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.400785    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" podStartSLOduration=2.400759978 podStartE2EDuration="2.400759978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.394943587 +0000 UTC m=+1954.741927490" watchObservedRunningTime="2026-10-31 00:15:02.400759978 +0000 UTC m=+1954.747743876"
Oct 31 00:15:02.619508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.619442    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" podStartSLOduration=2.619421849 podStartE2EDuration="2.619421849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.510492047 +0000 UTC m=+1954.857475944" watchObservedRunningTime="2026-10-31 00:15:02.619421849 +0000 UTC m=+1954.966405748"
Oct 31 00:15:02.619990 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.619960    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" podStartSLOduration=2.619948658 podStartE2EDuration="2.619948658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.618354523 +0000 UTC m=+1954.965338422" watchObservedRunningTime="2026-10-31 00:15:02.619948658 +0000 UTC m=+1954.966932557"
Oct 31 00:15:02.753052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.752988    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" podStartSLOduration=2.752973891 podStartE2EDuration="2.752973891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.750404565 +0000 UTC m=+1955.097388464" watchObservedRunningTime="2026-10-31 00:15:02.752973891 +0000 UTC m=+1955.099957789"
Oct 31 00:15:02.872189 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:02.872117    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" podStartSLOduration=2.872094414 podStartE2EDuration="2.872094414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:02.868338968 +0000 UTC m=+1955.215322898" watchObservedRunningTime="2026-10-31 00:15:02.872094414 +0000 UTC m=+1955.219078313"
Oct 31 00:15:03.085816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.085762    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" podStartSLOduration=3.085746912 podStartE2EDuration="3.085746912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:03.085605034 +0000 UTC m=+1955.432588933" watchObservedRunningTime="2026-10-31 00:15:03.085746912 +0000 UTC m=+1955.432730809"
Oct 31 00:15:03.198041 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.197965    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" podStartSLOduration=3.197945189 podStartE2EDuration="3.197945189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:03.195440724 +0000 UTC m=+1955.542424621" watchObservedRunningTime="2026-10-31 00:15:03.197945189 +0000 UTC m=+1955.544929088"
Oct 31 00:15:03.209697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.209660    6033 generic.go:356] "Generic (PLEG): container finished" podID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerID="366f7fa4d4ab9e2b4cf0b52dd2e677bc1dae56bba3d65f5e8c32a5444888c80f" exitCode=60
Oct 31 00:15:03.209901 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.209749    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerDied","Data":"366f7fa4d4ab9e2b4cf0b52dd2e677bc1dae56bba3d65f5e8c32a5444888c80f"}
Oct 31 00:15:03.210188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.210165    6033 scope.go:122] "RemoveContainer" containerID="366f7fa4d4ab9e2b4cf0b52dd2e677bc1dae56bba3d65f5e8c32a5444888c80f"
Oct 31 00:15:03.212503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.212476    6033 generic.go:356] "Generic (PLEG): container finished" podID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerID="1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b" exitCode=60
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.212568    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerDied","Data":"1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b"}
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.212661    6033 scope.go:122] "RemoveContainer" containerID="e0fe8709b2309220e0257b2183045a4255f00f9cfa1fa6abc68d172ae9ded728"
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.212858    6033 scope.go:122] "RemoveContainer" containerID="1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b"
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:03.213168    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v_odh-ai-gateway-infra(df8ef086-21f1-4eb8-88d0-0bb7f5a60db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4"
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.215363    6033 generic.go:356] "Generic (PLEG): container finished" podID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerID="7be47d5357a05afce559ec44add1c6bfe08962e0dae9c2a1fd51bd8346b8888c" exitCode=60
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.215446    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerDied","Data":"7be47d5357a05afce559ec44add1c6bfe08962e0dae9c2a1fd51bd8346b8888c"}
Oct 31 00:15:03.216477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.215839    6033 scope.go:122] "RemoveContainer" containerID="7be47d5357a05afce559ec44add1c6bfe08962e0dae9c2a1fd51bd8346b8888c"
Oct 31 00:15:03.219555 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.219528    6033 generic.go:356] "Generic (PLEG): container finished" podID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerID="b368c212c9e613e42818094d86c18cebf8420aba2c7127fbfbd4c05cd7b44e57" exitCode=60
Oct 31 00:15:03.219700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.219613    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerDied","Data":"b368c212c9e613e42818094d86c18cebf8420aba2c7127fbfbd4c05cd7b44e57"}
Oct 31 00:15:03.220095 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.220070    6033 scope.go:122] "RemoveContainer" containerID="b368c212c9e613e42818094d86c18cebf8420aba2c7127fbfbd4c05cd7b44e57"
Oct 31 00:15:03.221689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.221664    6033 generic.go:356] "Generic (PLEG): container finished" podID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerID="6f908a75c28047114ca76f5239911263d44b1cc7125debc74ba25ca47a4090dd" exitCode=60
Oct 31 00:15:03.221815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.221746    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerDied","Data":"6f908a75c28047114ca76f5239911263d44b1cc7125debc74ba25ca47a4090dd"}
Oct 31 00:15:03.222070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.222048    6033 scope.go:122] "RemoveContainer" containerID="6f908a75c28047114ca76f5239911263d44b1cc7125debc74ba25ca47a4090dd"
Oct 31 00:15:03.223719 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.223696    6033 generic.go:356] "Generic (PLEG): container finished" podID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerID="3cdd6e27a8bd5fff34b759d1ae7a56c7d69886c2f351f90539568f8bd1cdf911" exitCode=60
Oct 31 00:15:03.223832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.223777    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerDied","Data":"3cdd6e27a8bd5fff34b759d1ae7a56c7d69886c2f351f90539568f8bd1cdf911"}
Oct 31 00:15:03.224073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.224046    6033 scope.go:122] "RemoveContainer" containerID="3cdd6e27a8bd5fff34b759d1ae7a56c7d69886c2f351f90539568f8bd1cdf911"
Oct 31 00:15:03.226831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.226809    6033 generic.go:356] "Generic (PLEG): container finished" podID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerID="c41ad3e5d9c0d405ed3c795e61960299f3a644c7c680743cab7af3afdb3244d5" exitCode=60
Oct 31 00:15:03.227191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.227147    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerDied","Data":"c41ad3e5d9c0d405ed3c795e61960299f3a644c7c680743cab7af3afdb3244d5"}
Oct 31 00:15:03.227675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.227654    6033 scope.go:122] "RemoveContainer" containerID="c41ad3e5d9c0d405ed3c795e61960299f3a644c7c680743cab7af3afdb3244d5"
Oct 31 00:15:03.229582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.229561    6033 generic.go:356] "Generic (PLEG): container finished" podID="a6819373-9571-4d18-9412-951f6f930c13" containerID="fd856531a97a275e478bfd01462828e22401e649bade502603a61ef5f4734884" exitCode=60
Oct 31 00:15:03.229674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.229646    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerDied","Data":"fd856531a97a275e478bfd01462828e22401e649bade502603a61ef5f4734884"}
Oct 31 00:15:03.229990 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.229969    6033 scope.go:122] "RemoveContainer" containerID="fd856531a97a275e478bfd01462828e22401e649bade502603a61ef5f4734884"
Oct 31 00:15:03.231489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.231469    6033 generic.go:356] "Generic (PLEG): container finished" podID="29bfddbb-4c60-44a7-a122-4601fce98792" containerID="bfb694d94325d1613a90083efff796c7aea9c87aa0a397f05b95c712a170dd94" exitCode=60
Oct 31 00:15:03.231603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.231552    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerDied","Data":"bfb694d94325d1613a90083efff796c7aea9c87aa0a397f05b95c712a170dd94"}
Oct 31 00:15:03.231837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.231818    6033 scope.go:122] "RemoveContainer" containerID="bfb694d94325d1613a90083efff796c7aea9c87aa0a397f05b95c712a170dd94"
Oct 31 00:15:03.233219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.233195    6033 generic.go:356] "Generic (PLEG): container finished" podID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerID="ae06616c6d87a566a47c0d7ba21c03a93d58edc596b18e93baf0e85225c6040f" exitCode=60
Oct 31 00:15:03.233329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.233228    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerDied","Data":"ae06616c6d87a566a47c0d7ba21c03a93d58edc596b18e93baf0e85225c6040f"}
Oct 31 00:15:03.233567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.233549    6033 scope.go:122] "RemoveContainer" containerID="ae06616c6d87a566a47c0d7ba21c03a93d58edc596b18e93baf0e85225c6040f"
Oct 31 00:15:03.313876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:03.313611    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" podStartSLOduration=3.313571846 podStartE2EDuration="3.313571846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:03.311891388 +0000 UTC m=+1955.658875290" watchObservedRunningTime="2026-10-31 00:15:03.313571846 +0000 UTC m=+1955.660555745"
Oct 31 00:15:04.243774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.243694    6033 scope.go:122] "RemoveContainer" containerID="1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b"
Oct 31 00:15:04.244221 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:04.244160    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v_odh-ai-gateway-infra(df8ef086-21f1-4eb8-88d0-0bb7f5a60db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4"
Oct 31 00:15:04.245456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.245430    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerStarted","Data":"21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"}
Oct 31 00:15:04.247601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.247570    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerStarted","Data":"8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"}
Oct 31 00:15:04.249706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.249674    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerStarted","Data":"871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"}
Oct 31 00:15:04.251735 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.251707    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerStarted","Data":"d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"}
Oct 31 00:15:04.253909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.253878    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerStarted","Data":"42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"}
Oct 31 00:15:04.256205 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.256174    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerStarted","Data":"0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"}
Oct 31 00:15:04.258399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.258373    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerStarted","Data":"aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"}
Oct 31 00:15:04.260365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.260338    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerStarted","Data":"7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"}
Oct 31 00:15:04.262348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:04.262318    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerStarted","Data":"c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"}
Oct 31 00:15:05.274449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.274409    6033 generic.go:356] "Generic (PLEG): container finished" podID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerID="7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9" exitCode=60
Oct 31 00:15:05.274930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.274530    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerDied","Data":"7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"}
Oct 31 00:15:05.274930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.274587    6033 scope.go:122] "RemoveContainer" containerID="ae06616c6d87a566a47c0d7ba21c03a93d58edc596b18e93baf0e85225c6040f"
Oct 31 00:15:05.274930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.274846    6033 scope.go:122] "RemoveContainer" containerID="7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"
Oct 31 00:15:05.275381 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.275333    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8_odh-ai-gateway-infra(67bcc70c-7183-4e9b-ab35-1901b6ff6ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa"
Oct 31 00:15:05.277516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.277482    6033 generic.go:356] "Generic (PLEG): container finished" podID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerID="c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c" exitCode=60
Oct 31 00:15:05.277627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.277566    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerDied","Data":"c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"}
Oct 31 00:15:05.277870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.277855    6033 scope.go:122] "RemoveContainer" containerID="c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"
Oct 31 00:15:05.278241 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.278216    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx_odh-ai-gateway-infra(d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77"
Oct 31 00:15:05.279911 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.279878    6033 generic.go:356] "Generic (PLEG): container finished" podID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerID="21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4" exitCode=60
Oct 31 00:15:05.280061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.279954    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerDied","Data":"21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"}
Oct 31 00:15:05.280278 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.280263    6033 scope.go:122] "RemoveContainer" containerID="21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"
Oct 31 00:15:05.280645 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.280624    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84_odh-ai-gateway-infra(9bec1afd-a211-466b-9335-baaf35dd97a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3"
Oct 31 00:15:05.282160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.282134    6033 generic.go:356] "Generic (PLEG): container finished" podID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerID="8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29" exitCode=60
Oct 31 00:15:05.282266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.282236    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerDied","Data":"8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"}
Oct 31 00:15:05.282515 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.282499    6033 scope.go:122] "RemoveContainer" containerID="8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"
Oct 31 00:15:05.282843 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.282824    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-fh9pl_odh-ai-gateway-infra(c8e5b1b4-0064-4879-aeff-58065661ff23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23"
Oct 31 00:15:05.284436 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.284409    6033 generic.go:356] "Generic (PLEG): container finished" podID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerID="871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41" exitCode=60
Oct 31 00:15:05.284525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.284508    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerDied","Data":"871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"}
Oct 31 00:15:05.284769 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.284744    6033 scope.go:122] "RemoveContainer" containerID="871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"
Oct 31 00:15:05.285119 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.285099    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn_odh-ai-gateway-infra(7a7a22a7-3585-40b8-9516-aa6430aeb613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613"
Oct 31 00:15:05.286790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.286765    6033 generic.go:356] "Generic (PLEG): container finished" podID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerID="d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118" exitCode=60
Oct 31 00:15:05.286892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.286846    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerDied","Data":"d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"}
Oct 31 00:15:05.287116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.287097    6033 scope.go:122] "RemoveContainer" containerID="d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"
Oct 31 00:15:05.287469 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.287437    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5_odh-ai-gateway-infra(f14c80c7-70d7-4878-94c2-de9d7c7dd63e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e"
Oct 31 00:15:05.290258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.290233    6033 generic.go:356] "Generic (PLEG): container finished" podID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerID="42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3" exitCode=60
Oct 31 00:15:05.290394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.290313    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerDied","Data":"42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"}
Oct 31 00:15:05.290774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.290754    6033 scope.go:122] "RemoveContainer" containerID="42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"
Oct 31 00:15:05.291209 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.291183    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb_odh-ai-gateway-infra(ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe"
Oct 31 00:15:05.291897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.291878    6033 scope.go:122] "RemoveContainer" containerID="366f7fa4d4ab9e2b4cf0b52dd2e677bc1dae56bba3d65f5e8c32a5444888c80f"
Oct 31 00:15:05.292992 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.292974    6033 generic.go:356] "Generic (PLEG): container finished" podID="a6819373-9571-4d18-9412-951f6f930c13" containerID="0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171" exitCode=60
Oct 31 00:15:05.293317 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.293289    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerDied","Data":"0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"}
Oct 31 00:15:05.293672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.293652    6033 scope.go:122] "RemoveContainer" containerID="0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"
Oct 31 00:15:05.294073 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.294018    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn_odh-ai-gateway-infra(a6819373-9571-4d18-9412-951f6f930c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" podUID="a6819373-9571-4d18-9412-951f6f930c13"
Oct 31 00:15:05.295363 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.295345    6033 generic.go:356] "Generic (PLEG): container finished" podID="29bfddbb-4c60-44a7-a122-4601fce98792" containerID="aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1" exitCode=60
Oct 31 00:15:05.295439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.295370    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerDied","Data":"aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"}
Oct 31 00:15:05.295741 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.295721    6033 scope.go:122] "RemoveContainer" containerID="aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"
Oct 31 00:15:05.296100 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:05.296081    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv_odh-ai-gateway-infra(29bfddbb-4c60-44a7-a122-4601fce98792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" podUID="29bfddbb-4c60-44a7-a122-4601fce98792"
Oct 31 00:15:05.312245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.310654    6033 scope.go:122] "RemoveContainer" containerID="7be47d5357a05afce559ec44add1c6bfe08962e0dae9c2a1fd51bd8346b8888c"
Oct 31 00:15:05.329267 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.329237    6033 scope.go:122] "RemoveContainer" containerID="b368c212c9e613e42818094d86c18cebf8420aba2c7127fbfbd4c05cd7b44e57"
Oct 31 00:15:05.344654 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.344623    6033 scope.go:122] "RemoveContainer" containerID="6f908a75c28047114ca76f5239911263d44b1cc7125debc74ba25ca47a4090dd"
Oct 31 00:15:05.363823 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.363776    6033 scope.go:122] "RemoveContainer" containerID="3cdd6e27a8bd5fff34b759d1ae7a56c7d69886c2f351f90539568f8bd1cdf911"
Oct 31 00:15:05.382068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.382022    6033 scope.go:122] "RemoveContainer" containerID="c41ad3e5d9c0d405ed3c795e61960299f3a644c7c680743cab7af3afdb3244d5"
Oct 31 00:15:05.397699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.397668    6033 scope.go:122] "RemoveContainer" containerID="fd856531a97a275e478bfd01462828e22401e649bade502603a61ef5f4734884"
Oct 31 00:15:05.413383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:05.413335    6033 scope.go:122] "RemoveContainer" containerID="bfb694d94325d1613a90083efff796c7aea9c87aa0a397f05b95c712a170dd94"
Oct 31 00:15:06.307046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.307009    6033 scope.go:122] "RemoveContainer" containerID="42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"
Oct 31 00:15:06.307503 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.307462    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb_odh-ai-gateway-infra(ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe"
Oct 31 00:15:06.308910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.308886    6033 scope.go:122] "RemoveContainer" containerID="0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"
Oct 31 00:15:06.309320 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.309293    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn_odh-ai-gateway-infra(a6819373-9571-4d18-9412-951f6f930c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" podUID="a6819373-9571-4d18-9412-951f6f930c13"
Oct 31 00:15:06.310671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.310652    6033 scope.go:122] "RemoveContainer" containerID="aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"
Oct 31 00:15:06.311000 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.310979    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv_odh-ai-gateway-infra(29bfddbb-4c60-44a7-a122-4601fce98792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" podUID="29bfddbb-4c60-44a7-a122-4601fce98792"
Oct 31 00:15:06.312458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.312436    6033 scope.go:122] "RemoveContainer" containerID="7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"
Oct 31 00:15:06.312726 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.312701    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8_odh-ai-gateway-infra(67bcc70c-7183-4e9b-ab35-1901b6ff6ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa"
Oct 31 00:15:06.313926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.313899    6033 scope.go:122] "RemoveContainer" containerID="c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"
Oct 31 00:15:06.314223 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.314200    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx_odh-ai-gateway-infra(d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77"
Oct 31 00:15:06.315654 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.315621    6033 scope.go:122] "RemoveContainer" containerID="21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"
Oct 31 00:15:06.315890 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.315858    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84_odh-ai-gateway-infra(9bec1afd-a211-466b-9335-baaf35dd97a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3"
Oct 31 00:15:06.317524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.317496    6033 scope.go:122] "RemoveContainer" containerID="8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"
Oct 31 00:15:06.317751 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.317732    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-fh9pl_odh-ai-gateway-infra(c8e5b1b4-0064-4879-aeff-58065661ff23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23"
Oct 31 00:15:06.319172 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.319155    6033 scope.go:122] "RemoveContainer" containerID="871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"
Oct 31 00:15:06.319398 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.319373    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn_odh-ai-gateway-infra(7a7a22a7-3585-40b8-9516-aa6430aeb613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613"
Oct 31 00:15:06.320782 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:06.320764    6033 scope.go:122] "RemoveContainer" containerID="d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"
Oct 31 00:15:06.320961 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:06.320945    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5_odh-ai-gateway-infra(f14c80c7-70d7-4878-94c2-de9d7c7dd63e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e"
Oct 31 00:15:15.832234 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:15.832204    6033 scope.go:122] "RemoveContainer" containerID="0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"
Oct 31 00:15:15.832639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:15.832334    6033 scope.go:122] "RemoveContainer" containerID="871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"
Oct 31 00:15:15.832639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:15.832404    6033 scope.go:122] "RemoveContainer" containerID="c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"
Oct 31 00:15:16.402184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:16.402140    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerStarted","Data":"e75f80b8e79f033f4fe6631717a0ede7da98dc94acf62ab854f01b9e961806ee"}
Oct 31 00:15:16.404414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:16.404376    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerStarted","Data":"de7e4ea4dc66de2091c0931ba39f1233fbc036344f17cc5039ce355fc3c5d2ca"}
Oct 31 00:15:16.822733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:16.822699    6033 scope.go:122] "RemoveContainer" containerID="aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"
Oct 31 00:15:16.822893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:16.822840    6033 scope.go:122] "RemoveContainer" containerID="7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"
Oct 31 00:15:16.822966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:16.822951    6033 scope.go:122] "RemoveContainer" containerID="1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b"
Oct 31 00:15:16.971454 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:16.971414    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:15:17.425177 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.425134    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerStarted","Data":"8829cbb15dceb0c624d9ece9f391eaaa51f6a6e9082bbfae0fb0aaed6486fff4"}
Oct 31 00:15:17.427271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.427248    6033 generic.go:356] "Generic (PLEG): container finished" podID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerID="ff457d5d109143db9aa553a84d44e312f29d9c2a164217704a9306ae9dc7c3ff" exitCode=60
Oct 31 00:15:17.427404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.427318    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerDied","Data":"ff457d5d109143db9aa553a84d44e312f29d9c2a164217704a9306ae9dc7c3ff"}
Oct 31 00:15:17.427404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.427377    6033 scope.go:122] "RemoveContainer" containerID="c3bb006f34137fb5887e200bac4ff46908b7cde481407ac156a477f1380e0a0c"
Oct 31 00:15:17.427708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.427687    6033 scope.go:122] "RemoveContainer" containerID="ff457d5d109143db9aa553a84d44e312f29d9c2a164217704a9306ae9dc7c3ff"
Oct 31 00:15:17.428081 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:17.428021    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx_odh-ai-gateway-infra(d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77"
Oct 31 00:15:17.429644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.429613    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerStarted","Data":"a80d74c2b68dcd72718370f112065cc1705b4731d78c8128a523766f7beedc49"}
Oct 31 00:15:17.431434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.431412    6033 generic.go:356] "Generic (PLEG): container finished" podID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerID="e75f80b8e79f033f4fe6631717a0ede7da98dc94acf62ab854f01b9e961806ee" exitCode=60
Oct 31 00:15:17.431544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.431480    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerDied","Data":"e75f80b8e79f033f4fe6631717a0ede7da98dc94acf62ab854f01b9e961806ee"}
Oct 31 00:15:17.431807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.431793    6033 scope.go:122] "RemoveContainer" containerID="e75f80b8e79f033f4fe6631717a0ede7da98dc94acf62ab854f01b9e961806ee"
Oct 31 00:15:17.432162 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:17.432136    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn_odh-ai-gateway-infra(7a7a22a7-3585-40b8-9516-aa6430aeb613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613"
Oct 31 00:15:17.433403 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.433384    6033 generic.go:356] "Generic (PLEG): container finished" podID="a6819373-9571-4d18-9412-951f6f930c13" containerID="de7e4ea4dc66de2091c0931ba39f1233fbc036344f17cc5039ce355fc3c5d2ca" exitCode=60
Oct 31 00:15:17.433486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.433452    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerDied","Data":"de7e4ea4dc66de2091c0931ba39f1233fbc036344f17cc5039ce355fc3c5d2ca"}
Oct 31 00:15:17.433826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.433807    6033 scope.go:122] "RemoveContainer" containerID="de7e4ea4dc66de2091c0931ba39f1233fbc036344f17cc5039ce355fc3c5d2ca"
Oct 31 00:15:17.434130 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:17.434110    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn_odh-ai-gateway-infra(a6819373-9571-4d18-9412-951f6f930c13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" podUID="a6819373-9571-4d18-9412-951f6f930c13"
Oct 31 00:15:17.435633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.435614    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerStarted","Data":"0c8e7986f0d780403f572d640b814e1f9f458f9d92d18ab35f3bc5463684b03a"}
Oct 31 00:15:17.441679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.441658    6033 scope.go:122] "RemoveContainer" containerID="871cfab5bed782095a66194a91b266f4fe0588fd3bf4fb2822a5a353dce97a41"
Oct 31 00:15:17.456457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.456434    6033 scope.go:122] "RemoveContainer" containerID="0d6004f7d67137c67b396df6c1a7303b518227743fe8e0bcbc3597a45d0b2171"
Oct 31 00:15:17.521557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.521517    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:17.630446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.630410    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:17.822348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:17.822321    6033 scope.go:122] "RemoveContainer" containerID="42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"
Oct 31 00:15:18.094002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.093951    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" podStartSLOduration=18.093936211 podStartE2EDuration="18.093936211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:18.090768327 +0000 UTC m=+1970.437752226" watchObservedRunningTime="2026-10-31 00:15:18.093936211 +0000 UTC m=+1970.440920164"
Oct 31 00:15:18.446971 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.446870    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerStarted","Data":"e5e5adeeb248476d6e16775f420d86b11c67a4a221b8a222c2f7a12fcb320b8b"}
Oct 31 00:15:18.451478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.451450    6033 generic.go:356] "Generic (PLEG): container finished" podID="29bfddbb-4c60-44a7-a122-4601fce98792" containerID="0c8e7986f0d780403f572d640b814e1f9f458f9d92d18ab35f3bc5463684b03a" exitCode=60
Oct 31 00:15:18.451602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.451518    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerDied","Data":"0c8e7986f0d780403f572d640b814e1f9f458f9d92d18ab35f3bc5463684b03a"}
Oct 31 00:15:18.451602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.451576    6033 scope.go:122] "RemoveContainer" containerID="aebfd459a3a951547a6aa539637488e51a809506d2cbec2c761000ea24c3aaf1"
Oct 31 00:15:18.452068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.452004    6033 scope.go:122] "RemoveContainer" containerID="0c8e7986f0d780403f572d640b814e1f9f458f9d92d18ab35f3bc5463684b03a"
Oct 31 00:15:18.452551 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:18.452452    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv_odh-ai-gateway-infra(29bfddbb-4c60-44a7-a122-4601fce98792)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" podUID="29bfddbb-4c60-44a7-a122-4601fce98792"
Oct 31 00:15:18.453993 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.453972    6033 generic.go:356] "Generic (PLEG): container finished" podID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerID="8829cbb15dceb0c624d9ece9f391eaaa51f6a6e9082bbfae0fb0aaed6486fff4" exitCode=60
Oct 31 00:15:18.454114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.454061    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerDied","Data":"8829cbb15dceb0c624d9ece9f391eaaa51f6a6e9082bbfae0fb0aaed6486fff4"}
Oct 31 00:15:18.454470 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.454452    6033 scope.go:122] "RemoveContainer" containerID="8829cbb15dceb0c624d9ece9f391eaaa51f6a6e9082bbfae0fb0aaed6486fff4"
Oct 31 00:15:18.454810 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:18.454788    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8_odh-ai-gateway-infra(67bcc70c-7183-4e9b-ab35-1901b6ff6ffa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa"
Oct 31 00:15:18.458755 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.458716    6033 generic.go:356] "Generic (PLEG): container finished" podID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerID="a80d74c2b68dcd72718370f112065cc1705b4731d78c8128a523766f7beedc49" exitCode=60
Oct 31 00:15:18.458878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.458765    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerDied","Data":"a80d74c2b68dcd72718370f112065cc1705b4731d78c8128a523766f7beedc49"}
Oct 31 00:15:18.459313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.459291    6033 scope.go:122] "RemoveContainer" containerID="a80d74c2b68dcd72718370f112065cc1705b4731d78c8128a523766f7beedc49"
Oct 31 00:15:18.459741 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:18.459713    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v_odh-ai-gateway-infra(df8ef086-21f1-4eb8-88d0-0bb7f5a60db4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4"
Oct 31 00:15:18.472408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.467395    6033 scope.go:122] "RemoveContainer" containerID="7d54bcef203c4bcce4e89eb7de50c0f124d24bce0b781f3ddf099c4a673fb8b9"
Oct 31 00:15:18.491339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.489158    6033 scope.go:122] "RemoveContainer" containerID="1293d8c8528f0af1b67b86ae70bcd98194a92691bbe89ca3e6d9f8f87536612b"
Oct 31 00:15:18.548207 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.548080    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:18.654760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.654728    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:18.656744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.656725    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:18.764663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.764617    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:18.822088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.822020    6033 scope.go:122] "RemoveContainer" containerID="8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"
Oct 31 00:15:18.822248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.822108    6033 scope.go:122] "RemoveContainer" containerID="21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"
Oct 31 00:15:18.822248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.822206    6033 scope.go:122] "RemoveContainer" containerID="d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"
Oct 31 00:15:18.823469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.823437    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\") pod \"a6819373-9571-4d18-9412-951f6f930c13\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") "
Oct 31 00:15:18.823589 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.823499    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\" (UniqueName: \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\") pod \"7a7a22a7-3585-40b8-9516-aa6430aeb613\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") "
Oct 31 00:15:18.823589 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.823582    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\" (UniqueName: \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\") pod \"a6819373-9571-4d18-9412-951f6f930c13\" (UID: \"a6819373-9571-4d18-9412-951f6f930c13\") "
Oct 31 00:15:18.823702 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.823617    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\") pod \"7a7a22a7-3585-40b8-9516-aa6430aeb613\" (UID: \"7a7a22a7-3585-40b8-9516-aa6430aeb613\") "
Oct 31 00:15:18.824063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.823865    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca" pod "a6819373-9571-4d18-9412-951f6f930c13" (UID: "a6819373-9571-4d18-9412-951f6f930c13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:18.824341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.824197    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca" pod "7a7a22a7-3585-40b8-9516-aa6430aeb613" (UID: "7a7a22a7-3585-40b8-9516-aa6430aeb613"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:18.824489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.824465    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6819373-9571-4d18-9412-951f6f930c13-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.824550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.824494    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7a22a7-3585-40b8-9516-aa6430aeb613-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.826391 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.826366    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h" pod "7a7a22a7-3585-40b8-9516-aa6430aeb613" (UID: "7a7a22a7-3585-40b8-9516-aa6430aeb613"). InnerVolumeSpecName "kube-api-access-7hp7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:18.826504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.826478    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6" pod "a6819373-9571-4d18-9412-951f6f930c13" (UID: "a6819373-9571-4d18-9412-951f6f930c13"). InnerVolumeSpecName "kube-api-access-lb8t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:18.887313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.887233    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:18.926397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.926307    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hp7h\" (UniqueName: \"kubernetes.io/projected/7a7a22a7-3585-40b8-9516-aa6430aeb613-kube-api-access-7hp7h\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:18.926397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:18.926345    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8t6\" (UniqueName: \"kubernetes.io/projected/a6819373-9571-4d18-9412-951f6f930c13-kube-api-access-lb8t6\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.102820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.101232    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:19.170653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.170624    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:19.230454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.230414    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\" (UniqueName: \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\") pod \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") "
Oct 31 00:15:19.230454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.230496    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\") pod \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\" (UID: \"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77\") "
Oct 31 00:15:19.231298 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.231264    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca" pod "d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" (UID: "d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:19.233358 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.233328    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22" pod "d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" (UID: "d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77"). InnerVolumeSpecName "kube-api-access-8xk22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:19.332479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.332441    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk22\" (UniqueName: \"kubernetes.io/projected/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-kube-api-access-8xk22\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.332479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.332472    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.475152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.475119    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerStarted","Data":"36faddf2af24af8ee4bce4725ad79c1ebf8e09efa98ae169ab1e8d093aba39f9"}
Oct 31 00:15:19.477547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.477515    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerStarted","Data":"a826ab1c1e1c0bed5aebce5a40192f1b24ba24980e3d88be4e8d34e4de730af5"}
Oct 31 00:15:19.480323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.480171    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn" event={"ID":"7a7a22a7-3585-40b8-9516-aa6430aeb613","Type":"ContainerDied","Data":"9756669b0fb5308161cc5bd9a266dccd698161c9a6d7053206f0df422aec4faa"}
Oct 31 00:15:19.480323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.480237    6033 scope.go:122] "RemoveContainer" containerID="e75f80b8e79f033f4fe6631717a0ede7da98dc94acf62ab854f01b9e961806ee"
Oct 31 00:15:19.480323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.480179    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"
Oct 31 00:15:19.484210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.484168    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerStarted","Data":"4c54d27f223ce73f099cf2d26d62c3153553965d78359d5f6e6d1faf6a28dfba"}
Oct 31 00:15:19.487877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.487825    6033 generic.go:356] "Generic (PLEG): container finished" podID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerID="e5e5adeeb248476d6e16775f420d86b11c67a4a221b8a222c2f7a12fcb320b8b" exitCode=60
Oct 31 00:15:19.488099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.487868    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerDied","Data":"e5e5adeeb248476d6e16775f420d86b11c67a4a221b8a222c2f7a12fcb320b8b"}
Oct 31 00:15:19.488389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.488366    6033 scope.go:122] "RemoveContainer" containerID="e5e5adeeb248476d6e16775f420d86b11c67a4a221b8a222c2f7a12fcb320b8b"
Oct 31 00:15:19.489613 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:19.488757    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb_odh-ai-gateway-infra(ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe"
Oct 31 00:15:19.489957 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.489931    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"
Oct 31 00:15:19.493308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.489933    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn" event={"ID":"a6819373-9571-4d18-9412-951f6f930c13","Type":"ContainerDied","Data":"e962795a7fb7538565e0331624660193dfcadd881c222dca92fe30ef975195c3"}
Oct 31 00:15:19.500218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.500195    6033 scope.go:122] "RemoveContainer" containerID="42fbd86d45c156411675ac8a10a33641324c19dd021abe25c599906054a1eea3"
Oct 31 00:15:19.506137 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.506083    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx" event={"ID":"d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77","Type":"ContainerDied","Data":"aa948befa7d7271ffb3f865c5542dcc615eeaae98a6b011df1e2834d7fdb0724"}
Oct 31 00:15:19.506271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.506206    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"
Oct 31 00:15:19.542281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.542009    6033 scope.go:122] "RemoveContainer" containerID="de7e4ea4dc66de2091c0931ba39f1233fbc036344f17cc5039ce355fc3c5d2ca"
Oct 31 00:15:19.558334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.558250    6033 scope.go:122] "RemoveContainer" containerID="ff457d5d109143db9aa553a84d44e312f29d9c2a164217704a9306ae9dc7c3ff"
Oct 31 00:15:19.582264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.582222    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:19.698199 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.698164    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:19.721274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.720961    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:19.730003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.729975    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:19.738890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.738859    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\" (UniqueName: \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\") pod \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") "
Oct 31 00:15:19.739102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.738923    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\") pod \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\" (UID: \"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4\") "
Oct 31 00:15:19.739102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.738988    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\" (UniqueName: \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\") pod \"29bfddbb-4c60-44a7-a122-4601fce98792\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") "
Oct 31 00:15:19.739102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739050    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\" (UniqueName: \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\") pod \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") "
Oct 31 00:15:19.739102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739092    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\") pod \"29bfddbb-4c60-44a7-a122-4601fce98792\" (UID: \"29bfddbb-4c60-44a7-a122-4601fce98792\") "
Oct 31 00:15:19.739329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739286    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\") pod \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\" (UID: \"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa\") "
Oct 31 00:15:19.739499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739474    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca" pod "df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" (UID: "df8ef086-21f1-4eb8-88d0-0bb7f5a60db4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:19.739701 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739679    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca" pod "67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" (UID: "67bcc70c-7183-4e9b-ab35-1901b6ff6ffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:19.739827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739810    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.739877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739822    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca" pod "29bfddbb-4c60-44a7-a122-4601fce98792" (UID: "29bfddbb-4c60-44a7-a122-4601fce98792"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:19.739877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.739832    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.741430 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.741399    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd" pod "29bfddbb-4c60-44a7-a122-4601fce98792" (UID: "29bfddbb-4c60-44a7-a122-4601fce98792"). InnerVolumeSpecName "kube-api-access-qgvbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:19.741598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.741578    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4" pod "67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" (UID: "67bcc70c-7183-4e9b-ab35-1901b6ff6ffa"). InnerVolumeSpecName "kube-api-access-j6xl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:19.742620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.742600    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz" pod "df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" (UID: "df8ef086-21f1-4eb8-88d0-0bb7f5a60db4"). InnerVolumeSpecName "kube-api-access-khljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:19.840933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.840892    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khljz\" (UniqueName: \"kubernetes.io/projected/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4-kube-api-access-khljz\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.840933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.840932    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgvbd\" (UniqueName: \"kubernetes.io/projected/29bfddbb-4c60-44a7-a122-4601fce98792-kube-api-access-qgvbd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.841148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.840945    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xl4\" (UniqueName: \"kubernetes.io/projected/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa-kube-api-access-j6xl4\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.841148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.840959    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfddbb-4c60-44a7-a122-4601fce98792-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:19.994508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.994473    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:19.998353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:19.998312    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-d5360e-29890095-ccwcn"]
Oct 31 00:15:20.212889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.212773    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:20.218493 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.218243    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71b76a-29890095-w4lnx"]
Oct 31 00:15:20.518667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.518640    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"
Oct 31 00:15:20.518845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.518680    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv" event={"ID":"29bfddbb-4c60-44a7-a122-4601fce98792","Type":"ContainerDied","Data":"2fc7fb6fe952e1b1f9959d89239a9da91a27ca4af9333d127a8a37c817cbb61f"}
Oct 31 00:15:20.518845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.518743    6033 scope.go:122] "RemoveContainer" containerID="0c8e7986f0d780403f572d640b814e1f9f458f9d92d18ab35f3bc5463684b03a"
Oct 31 00:15:20.520736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.520697    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:20.520850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.520822    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"
Oct 31 00:15:20.520915 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.520848    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8" event={"ID":"67bcc70c-7183-4e9b-ab35-1901b6ff6ffa","Type":"ContainerDied","Data":"19fa2102d6d39cbb2b59321cb6ee14d76666109ce2ad1b7af84ac313e52b3daf"}
Oct 31 00:15:20.525799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.525772    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"
Oct 31 00:15:20.526204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.526171    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v" event={"ID":"df8ef086-21f1-4eb8-88d0-0bb7f5a60db4","Type":"ContainerDied","Data":"e0d3e67afeccf0a46a2561867d801e125293cca838f59f1a0d004843cc29b425"}
Oct 31 00:15:20.529291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.529259    6033 generic.go:356] "Generic (PLEG): container finished" podID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerID="36faddf2af24af8ee4bce4725ad79c1ebf8e09efa98ae169ab1e8d093aba39f9" exitCode=60
Oct 31 00:15:20.529435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.529352    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerDied","Data":"36faddf2af24af8ee4bce4725ad79c1ebf8e09efa98ae169ab1e8d093aba39f9"}
Oct 31 00:15:20.529875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.529851    6033 scope.go:122] "RemoveContainer" containerID="36faddf2af24af8ee4bce4725ad79c1ebf8e09efa98ae169ab1e8d093aba39f9"
Oct 31 00:15:20.530470 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:20.530439    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84_odh-ai-gateway-infra(9bec1afd-a211-466b-9335-baaf35dd97a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3"
Oct 31 00:15:20.531845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.531823    6033 generic.go:356] "Generic (PLEG): container finished" podID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerID="a826ab1c1e1c0bed5aebce5a40192f1b24ba24980e3d88be4e8d34e4de730af5" exitCode=60
Oct 31 00:15:20.531968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.531898    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerDied","Data":"a826ab1c1e1c0bed5aebce5a40192f1b24ba24980e3d88be4e8d34e4de730af5"}
Oct 31 00:15:20.532504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.532459    6033 scope.go:122] "RemoveContainer" containerID="a826ab1c1e1c0bed5aebce5a40192f1b24ba24980e3d88be4e8d34e4de730af5"
Oct 31 00:15:20.532932 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:20.532898    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-fh9pl_odh-ai-gateway-infra(c8e5b1b4-0064-4879-aeff-58065661ff23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23"
Oct 31 00:15:20.534173 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.534149    6033 scope.go:122] "RemoveContainer" containerID="8829cbb15dceb0c624d9ece9f391eaaa51f6a6e9082bbfae0fb0aaed6486fff4"
Oct 31 00:15:20.537126 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.537095    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f7b8d8-29890095-rt8dn"]
Oct 31 00:15:20.537893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.537528    6033 generic.go:356] "Generic (PLEG): container finished" podID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerID="4c54d27f223ce73f099cf2d26d62c3153553965d78359d5f6e6d1faf6a28dfba" exitCode=60
Oct 31 00:15:20.537893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.537571    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerDied","Data":"4c54d27f223ce73f099cf2d26d62c3153553965d78359d5f6e6d1faf6a28dfba"}
Oct 31 00:15:20.538102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.538080    6033 scope.go:122] "RemoveContainer" containerID="4c54d27f223ce73f099cf2d26d62c3153553965d78359d5f6e6d1faf6a28dfba"
Oct 31 00:15:20.538547 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:20.538522    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5_odh-ai-gateway-infra(f14c80c7-70d7-4878-94c2-de9d7c7dd63e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e"
Oct 31 00:15:20.548700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.548677    6033 scope.go:122] "RemoveContainer" containerID="a80d74c2b68dcd72718370f112065cc1705b4731d78c8128a523766f7beedc49"
Oct 31 00:15:20.563473 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.563437    6033 scope.go:122] "RemoveContainer" containerID="21b255b20462555a0384c99fcdd284609ccbfa705c625f93b94ac2eadd74c1d4"
Oct 31 00:15:20.583216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.583190    6033 scope.go:122] "RemoveContainer" containerID="8442eab6bac4cd63385e7d6f0654e3d9a99279ab710b1d8f1bd1e2b1a7193a29"
Oct 31 00:15:20.602253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.601426    6033 scope.go:122] "RemoveContainer" containerID="d01f46ceceb2f4b9201b1bd561bfb16233651a11bd22404d9166a33dd49b4118"
Oct 31 00:15:20.708302 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.708263    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:20.753040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.753003    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\") pod \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") "
Oct 31 00:15:20.753211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.753188    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\" (UniqueName: \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\") pod \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\" (UID: \"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe\") "
Oct 31 00:15:20.753494 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.753465    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca" pod "ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" (UID: "ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:20.753698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.753681    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:20.755514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.755473    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh" pod "ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" (UID: "ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe"). InnerVolumeSpecName "kube-api-access-cwhsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:20.855085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.854975    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhsh\" (UniqueName: \"kubernetes.io/projected/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe-kube-api-access-cwhsh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:20.897850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.897810    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:20.922466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.922425    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:20.927950 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:20.927914    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-54ad8c-29890095-nbq4v"]
Oct 31 00:15:21.081476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.081440    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:15:21.083237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083191    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.083237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083223    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.083237 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083239    6033 cpu_manager.go:414] "containerMap: removing container" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083247    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083268    6033 cpu_manager.go:414] "containerMap: removing container" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083275    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083290    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083298    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083312    6033 cpu_manager.go:414] "containerMap: removing container" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083319    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083329    6033 cpu_manager.go:414] "containerMap: removing container" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083336    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083346    6033 cpu_manager.go:414] "containerMap: removing container" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083354    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083363    6033 cpu_manager.go:414] "containerMap: removing container" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083370    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083381    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083386    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083402    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083407    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083416    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083423    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083429    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083438    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083451    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.083545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083456    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083942    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083958    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083971    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.083975    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084003    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084009    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084040    6033 cpu_manager.go:414] "containerMap: removing container" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084048    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084069    6033 cpu_manager.go:414] "containerMap: removing container" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084145    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084171    6033 cpu_manager.go:414] "containerMap: removing container" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084179    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084314    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084328    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084339    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084348    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084358    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084368    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084378    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084385    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084394    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084405    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084417    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084425    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084435    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084446    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:21.084722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.084477    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:21.088722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.088694    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.141409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.141287    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-348a33-serving-cert\""
Oct 31 00:15:21.145641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.145582    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:15:21.160487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.158670    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.160487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.158770    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.160487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.158808    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8shs\" (UniqueName: \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.160487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.158859    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.260471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.260423    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.261408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.260549    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.261408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.260599    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8shs\" (UniqueName: \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.261408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.260661    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.261666 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:21.261561    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-serving-cert: secret "maas-api-e2e-inf-a-348a33-serving-cert" not found
Oct 31 00:15:21.261721 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:21.261671    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls podName:364c444f-bc4e-4d7c-8112-00e0fdf56bcb nodeName:}" failed. No retries permitted until 2026-10-31 00:15:21.761647739 +0000 UTC m=+1974.108631632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls") pod "maas-api-e2e-inf-a-348a33-586dcb694-bldzw" (UID: "364c444f-bc4e-4d7c-8112-00e0fdf56bcb") : secret "maas-api-e2e-inf-a-348a33-serving-cert" not found
Oct 31 00:15:21.263176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.263146    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.264498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.264469    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.316923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.315098    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:21.331663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.331623    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8shs\" (UniqueName: \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.340921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.340884    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab0ac9-29890095-h4mrv"]
Oct 31 00:15:21.379845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.379812    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:21.556156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.556130    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"
Oct 31 00:15:21.556353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.556318    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb" event={"ID":"ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe","Type":"ContainerDied","Data":"bb2c290869bd9a03b72a7405abae896244daa102439d61f0237128ccae3f7c50"}
Oct 31 00:15:21.556564 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.556546    6033 scope.go:122] "RemoveContainer" containerID="e5e5adeeb248476d6e16775f420d86b11c67a4a221b8a222c2f7a12fcb320b8b"
Oct 31 00:15:21.661445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.661409    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:21.728703 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.728669    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:21.733979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.733939    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5c3374-29890095-c26f8"]
Oct 31 00:15:21.768435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.768370    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.776091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.774678    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") pod \"maas-api-e2e-inf-a-348a33-586dcb694-bldzw\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:21.797098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.797066    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:21.801736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.801662    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:21.832253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.832215    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" path="/var/lib/kubelet/pods/29bfddbb-4c60-44a7-a122-4601fce98792/volumes"
Oct 31 00:15:21.832727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.832709    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" path="/var/lib/kubelet/pods/67bcc70c-7183-4e9b-ab35-1901b6ff6ffa/volumes"
Oct 31 00:15:21.833175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.833157    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" path="/var/lib/kubelet/pods/7a7a22a7-3585-40b8-9516-aa6430aeb613/volumes"
Oct 31 00:15:21.833977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.833952    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6819373-9571-4d18-9412-951f6f930c13" path="/var/lib/kubelet/pods/a6819373-9571-4d18-9412-951f6f930c13/volumes"
Oct 31 00:15:21.834436 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.834415    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" path="/var/lib/kubelet/pods/d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77/volumes"
Oct 31 00:15:21.834961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.834940    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" path="/var/lib/kubelet/pods/df8ef086-21f1-4eb8-88d0-0bb7f5a60db4/volumes"
Oct 31 00:15:21.864557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.864526    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:21.869438 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.869408    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\") pod \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") "
Oct 31 00:15:21.869606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.869587    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\" (UniqueName: \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\") pod \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\" (UID: \"f14c80c7-70d7-4878-94c2-de9d7c7dd63e\") "
Oct 31 00:15:21.869765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.869752    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\" (UniqueName: \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\") pod \"c8e5b1b4-0064-4879-aeff-58065661ff23\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") "
Oct 31 00:15:21.869833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.869787    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\") pod \"c8e5b1b4-0064-4879-aeff-58065661ff23\" (UID: \"c8e5b1b4-0064-4879-aeff-58065661ff23\") "
Oct 31 00:15:21.869895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.869843    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca" pod "f14c80c7-70d7-4878-94c2-de9d7c7dd63e" (UID: "f14c80c7-70d7-4878-94c2-de9d7c7dd63e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:21.870265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.870238    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca" pod "c8e5b1b4-0064-4879-aeff-58065661ff23" (UID: "c8e5b1b4-0064-4879-aeff-58065661ff23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:21.870357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.870284    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.871794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.871765    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p" pod "f14c80c7-70d7-4878-94c2-de9d7c7dd63e" (UID: "f14c80c7-70d7-4878-94c2-de9d7c7dd63e"). InnerVolumeSpecName "kube-api-access-p8j7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:21.871925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.871907    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf" pod "c8e5b1b4-0064-4879-aeff-58065661ff23" (UID: "c8e5b1b4-0064-4879-aeff-58065661ff23"). InnerVolumeSpecName "kube-api-access-xp6kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:21.971387 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971348    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\") pod \"9bec1afd-a211-466b-9335-baaf35dd97a3\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") "
Oct 31 00:15:21.971542 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971498    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\" (UniqueName: \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\") pod \"9bec1afd-a211-466b-9335-baaf35dd97a3\" (UID: \"9bec1afd-a211-466b-9335-baaf35dd97a3\") "
Oct 31 00:15:21.971766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971741    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca" pod "9bec1afd-a211-466b-9335-baaf35dd97a3" (UID: "9bec1afd-a211-466b-9335-baaf35dd97a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:21.971975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971951    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bec1afd-a211-466b-9335-baaf35dd97a3-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.971975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971968    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp6kf\" (UniqueName: \"kubernetes.io/projected/c8e5b1b4-0064-4879-aeff-58065661ff23-kube-api-access-xp6kf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.971975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971979    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e5b1b4-0064-4879-aeff-58065661ff23-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.972198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.971988    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8j7p\" (UniqueName: \"kubernetes.io/projected/f14c80c7-70d7-4878-94c2-de9d7c7dd63e-kube-api-access-p8j7p\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:21.973597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:21.973573    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9" pod "9bec1afd-a211-466b-9335-baaf35dd97a3" (UID: "9bec1afd-a211-466b-9335-baaf35dd97a3"). InnerVolumeSpecName "kube-api-access-cpdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.015268 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.015232    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:22.076303 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.075795    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdn9\" (UniqueName: \"kubernetes.io/projected/9bec1afd-a211-466b-9335-baaf35dd97a3-kube-api-access-cpdn9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.180402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.180366    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:22.184153 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.184117    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0d75be-29890095-qbwfb"]
Oct 31 00:15:22.441096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.441056    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:15:22.446556 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.446528    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:15:22.575331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.575301    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"
Oct 31 00:15:22.575613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.575298    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84" event={"ID":"9bec1afd-a211-466b-9335-baaf35dd97a3","Type":"ContainerDied","Data":"544ab11ac15160bc754eee1cf73fff0977f41ae9bf86b945720204ea94c1c9fc"}
Oct 31 00:15:22.575730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.575682    6033 scope.go:122] "RemoveContainer" containerID="36faddf2af24af8ee4bce4725ad79c1ebf8e09efa98ae169ab1e8d093aba39f9"
Oct 31 00:15:22.577822 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.577803    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"
Oct 31 00:15:22.578139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.577802    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl" event={"ID":"c8e5b1b4-0064-4879-aeff-58065661ff23","Type":"ContainerDied","Data":"8165522a9a8542d1201504a30578212e9fff40b537ac2d3267509ad083222058"}
Oct 31 00:15:22.584015 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.580192    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5" event={"ID":"f14c80c7-70d7-4878-94c2-de9d7c7dd63e","Type":"ContainerDied","Data":"d2fd25c30277edd204293b43edf93145af616fdc7ffbc25d1ed6d55dac511887"}
Oct 31 00:15:22.584015 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.580307    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"
Oct 31 00:15:22.589123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.589102    6033 scope.go:122] "RemoveContainer" containerID="a826ab1c1e1c0bed5aebce5a40192f1b24ba24980e3d88be4e8d34e4de730af5"
Oct 31 00:15:22.590806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.590777    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" event={"ID":"364c444f-bc4e-4d7c-8112-00e0fdf56bcb","Type":"ContainerStarted","Data":"9fec64150ab46179177f3339b4f2f99e8aaef90305b970d7c1d9f947c50ff827"}
Oct 31 00:15:22.600067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.600040    6033 scope.go:122] "RemoveContainer" containerID="4c54d27f223ce73f099cf2d26d62c3153553965d78359d5f6e6d1faf6a28dfba"
Oct 31 00:15:22.752038 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.751997    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:22.755790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.755741    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-fh9pl"]
Oct 31 00:15:22.858782 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.858738    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:22.863447 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.863411    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03c703-29890095-bms84"]
Oct 31 00:15:22.969566 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.969527    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:22.975112 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:22.975077    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-35a187-29890095-dmjx5"]
Oct 31 00:15:23.619662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.619625    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" event={"ID":"364c444f-bc4e-4d7c-8112-00e0fdf56bcb","Type":"ContainerStarted","Data":"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"}
Oct 31 00:15:23.620132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.619735    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:23.732226 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.732113    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" podStartSLOduration=1.7883278360000001 podStartE2EDuration="2.732097691s" totalImagesPullingTime="943.769855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:23.730765768 +0000 UTC m=+1976.077749666" watchObservedRunningTime="2026-10-31 00:15:23.732097691 +0000 UTC m=+1976.079081589"
Oct 31 00:15:23.831699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.831665    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" path="/var/lib/kubelet/pods/9bec1afd-a211-466b-9335-baaf35dd97a3/volumes"
Oct 31 00:15:23.832170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.832155    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" path="/var/lib/kubelet/pods/c8e5b1b4-0064-4879-aeff-58065661ff23/volumes"
Oct 31 00:15:23.832574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.832560    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" path="/var/lib/kubelet/pods/ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe/volumes"
Oct 31 00:15:23.833301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:23.833285    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" path="/var/lib/kubelet/pods/f14c80c7-70d7-4878-94c2-de9d7c7dd63e/volumes"
Oct 31 00:15:25.841596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.841553    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:15:25.842572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842545    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.842572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842572    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842605    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842613    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842631    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842642    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842653    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842660    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842677    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842684    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6819373-9571-4d18-9412-951f6f930c13" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842692    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842697    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842712    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842716    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842726    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:25.842763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842731    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842871    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842879    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bcc70c-7183-4e9b-ab35-1901b6ff6ffa" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842886    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842893    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842905    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842914    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8ef086-21f1-4eb8-88d0-0bb7f5a60db4" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842919    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="29bfddbb-4c60-44a7-a122-4601fce98792" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842926    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842932    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbd5f46-afd9-4ab7-937e-cc3d5ac0cffe" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842939    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7a22a7-3585-40b8-9516-aa6430aeb613" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842943    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:25.843425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.842948    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b5fb71-afe0-4f1a-8a4e-5baa07a5ff77" containerName="cleanup"
Oct 31 00:15:25.848545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.848508    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:25.898135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.898099    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a251fdfb-serving-cert\""
Oct 31 00:15:25.944372 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.944332    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:25.944641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.944600    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:25.944721 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.944676    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64jq\" (UniqueName: \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:25.944778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.944751    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:25.949875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:25.949838    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:15:26.046273 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.046225    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.046488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.046440    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.046557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.046493    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64jq\" (UniqueName: \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.046607 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.046552    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.046712 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:26.046694    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-serving-cert: secret "maas-api-e2e-mt-a251fdfb-serving-cert" not found
Oct 31 00:15:26.046774 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:26.046759    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls podName:271e1e35-91a4-4c9e-9440-3e2e6d0257a8 nodeName:}" failed. No retries permitted until 2026-10-31 00:15:26.546740371 +0000 UTC m=+1978.893724252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls") pod "maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" (UID: "271e1e35-91a4-4c9e-9440-3e2e6d0257a8") : secret "maas-api-e2e-mt-a251fdfb-serving-cert" not found
Oct 31 00:15:26.046910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.046887    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.048954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.048936    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.117651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.117566    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64jq\" (UniqueName: \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.554017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.553966    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.556600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.556568    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") pod \"maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:26.764872 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:26.764830    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:27.155303 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:27.155272    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:15:27.157728 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:27.157695    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271e1e35_91a4_4c9e_9440_3e2e6d0257a8.slice/crio-1909c3096db5b0629087c81e4621f5e5ad8c116f2e807913fea44f4321f2cad9 WatchSource:0}: Error finding container 1909c3096db5b0629087c81e4621f5e5ad8c116f2e807913fea44f4321f2cad9: Status 404 returned error can't find the container with id 1909c3096db5b0629087c81e4621f5e5ad8c116f2e807913fea44f4321f2cad9
Oct 31 00:15:27.659489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:27.659443    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" event={"ID":"271e1e35-91a4-4c9e-9440-3e2e6d0257a8","Type":"ContainerStarted","Data":"1909c3096db5b0629087c81e4621f5e5ad8c116f2e807913fea44f4321f2cad9"}
Oct 31 00:15:28.669103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:28.669064    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" event={"ID":"271e1e35-91a4-4c9e-9440-3e2e6d0257a8","Type":"ContainerStarted","Data":"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"}
Oct 31 00:15:28.669615 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:28.669247    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:28.785467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:28.785408    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" podStartSLOduration=2.760425798 podStartE2EDuration="3.78539255s" totalImagesPullingTime="1.024966752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:28.782964584 +0000 UTC m=+1981.129948495" watchObservedRunningTime="2026-10-31 00:15:28.78539255 +0000 UTC m=+1981.132376448"
Oct 31 00:15:29.633313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:29.633281    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:15:34.682010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:34.681971    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:15:44.023668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.023579    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"]
Oct 31 00:15:44.024312 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024294    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:44.024402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024313    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:44.024402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024368    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:44.024402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024374    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:44.024402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024387    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:44.024402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024392    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:44.024598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024513    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8e5b1b4-0064-4879-aeff-58065661ff23" containerName="cleanup"
Oct 31 00:15:44.024598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024536    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14c80c7-70d7-4878-94c2-de9d7c7dd63e" containerName="cleanup"
Oct 31 00:15:44.024598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.024548    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bec1afd-a211-466b-9335-baaf35dd97a3" containerName="cleanup"
Oct 31 00:15:44.031188 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.031165    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.085889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.085854    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"]
Oct 31 00:15:44.128978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.128933    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdbw\" (UniqueName: \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.128978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.128972    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:15:44.129209 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.129019    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:15:44.129256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.129201    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.129373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.129353    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.230227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.230184    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.230442 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.230304    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.230521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.230442    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdbw\" (UniqueName: \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.230896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.230874    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.232817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.232793    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.288671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.288593    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdbw\" (UniqueName: \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.346245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.346207    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:44.720062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.720021    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"]
Oct 31 00:15:44.721881 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:15:44.721847    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4169271_f966_4c85_98b2_ac6adc9403f6.slice/crio-baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81 WatchSource:0}: Error finding container baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81: Status 404 returned error can't find the container with id baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81
Oct 31 00:15:44.787677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:44.787642    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc" event={"ID":"d4169271-f966-4c85-98b2-ac6adc9403f6","Type":"ContainerStarted","Data":"baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81"}
Oct 31 00:15:45.797869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:45.797831    6033 generic.go:356] "Generic (PLEG): container finished" podID="d4169271-f966-4c85-98b2-ac6adc9403f6" containerID="d5f1c5eee2576e105f26fa7e7340126e10734b612debf9a1c3472ce1e4974ea6" exitCode=0
Oct 31 00:15:45.798389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:45.797886    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc" event={"ID":"d4169271-f966-4c85-98b2-ac6adc9403f6","Type":"ContainerDied","Data":"d5f1c5eee2576e105f26fa7e7340126e10734b612debf9a1c3472ce1e4974ea6"}
Oct 31 00:15:46.980841 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:15:46.980804    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:15:46.985198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:46.985173    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:47.071069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.070933    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\") pod \"d4169271-f966-4c85-98b2-ac6adc9403f6\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") "
Oct 31 00:15:47.071069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.071010    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\" (UniqueName: \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\") pod \"d4169271-f966-4c85-98b2-ac6adc9403f6\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") "
Oct 31 00:15:47.071339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.071150    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\") pod \"d4169271-f966-4c85-98b2-ac6adc9403f6\" (UID: \"d4169271-f966-4c85-98b2-ac6adc9403f6\") "
Oct 31 00:15:47.071408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.071347    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca" pod "d4169271-f966-4c85-98b2-ac6adc9403f6" (UID: "d4169271-f966-4c85-98b2-ac6adc9403f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:47.071850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.071828    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:47.073405 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.073377    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw" pod "d4169271-f966-4c85-98b2-ac6adc9403f6" (UID: "d4169271-f966-4c85-98b2-ac6adc9403f6"). InnerVolumeSpecName "kube-api-access-4sdbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:47.073557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.073471    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca" pod "d4169271-f966-4c85-98b2-ac6adc9403f6" (UID: "d4169271-f966-4c85-98b2-ac6adc9403f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:15:47.173098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.173011    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdbw\" (UniqueName: \"kubernetes.io/projected/d4169271-f966-4c85-98b2-ac6adc9403f6-kube-api-access-4sdbw\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:47.173098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.173087    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4169271-f966-4c85-98b2-ac6adc9403f6-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:15:47.813644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.813614    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"
Oct 31 00:15:47.813835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.813612    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc" event={"ID":"d4169271-f966-4c85-98b2-ac6adc9403f6","Type":"ContainerDied","Data":"baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81"}
Oct 31 00:15:47.813835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:47.813736    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baf46f5379abf088c6e3e8b8258604ae658c0aa1ae3ab43ec983881c63bcff81"
Oct 31 00:15:50.357374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:15:50.357336    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:16:01.161672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:01.161630    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"]
Oct 31 00:16:01.204762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:01.204726    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-54ad8c-xsg6d"]
Oct 31 00:16:01.830422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:01.830383    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b13a87-fa22-4f30-a443-c893e9bf5f8e" path="/var/lib/kubelet/pods/33b13a87-fa22-4f30-a443-c893e9bf5f8e/volumes"
Oct 31 00:16:09.589522 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.589468    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"]
Oct 31 00:16:09.590394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.590367    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d4169271-f966-4c85-98b2-ac6adc9403f6" containerName="revoke-keys"
Oct 31 00:16:09.590394 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.590395    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4169271-f966-4c85-98b2-ac6adc9403f6" containerName="revoke-keys"
Oct 31 00:16:09.590624 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.590611    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4169271-f966-4c85-98b2-ac6adc9403f6" containerName="revoke-keys"
Oct 31 00:16:09.595875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.595849    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.643369 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.643320    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.643586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.643426    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscbb\" (UniqueName: \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.643666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.643601    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.652203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.652161    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"]
Oct 31 00:16:09.694414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.694382    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:16:09.694414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.694383    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:16:09.745400 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.745360    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.745400 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.745413    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscbb\" (UniqueName: \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.745689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.745528    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.746270 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.746246    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.748059 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.748017    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.803287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.803246    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscbb\" (UniqueName: \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:09.912540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:09.912426    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:10.295784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:10.295745    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"]
Oct 31 00:16:10.297805 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:10.297464    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa6042e_174b_46ac_af63_a59dc462776f.slice/crio-aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8 WatchSource:0}: Error finding container aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8: Status 404 returned error can't find the container with id aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8
Oct 31 00:16:11.081643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:11.081594    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k" event={"ID":"9fa6042e-174b-46ac-af63-a59dc462776f","Type":"ContainerStarted","Data":"ca66440fcd772b211edfe4a6b1bb0acdce423648ef19e65f58dfa4ba09705edf"}
Oct 31 00:16:11.082203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:11.081669    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k" event={"ID":"9fa6042e-174b-46ac-af63-a59dc462776f","Type":"ContainerStarted","Data":"aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8"}
Oct 31 00:16:11.194643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:11.194578    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k" podStartSLOduration=2.194561836 podStartE2EDuration="2.194561836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:11.192128302 +0000 UTC m=+2023.539112202" watchObservedRunningTime="2026-10-31 00:16:11.194561836 +0000 UTC m=+2023.541545770"
Oct 31 00:16:12.091424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:12.091388    6033 generic.go:356] "Generic (PLEG): container finished" podID="9fa6042e-174b-46ac-af63-a59dc462776f" containerID="ca66440fcd772b211edfe4a6b1bb0acdce423648ef19e65f58dfa4ba09705edf" exitCode=0
Oct 31 00:16:12.091892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:12.091467    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k" event={"ID":"9fa6042e-174b-46ac-af63-a59dc462776f","Type":"ContainerDied","Data":"ca66440fcd772b211edfe4a6b1bb0acdce423648ef19e65f58dfa4ba09705edf"}
Oct 31 00:16:13.277401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.277373    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:13.399862 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.399769    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\" (UniqueName: \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\") pod \"9fa6042e-174b-46ac-af63-a59dc462776f\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") "
Oct 31 00:16:13.400143 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.400120    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\") pod \"9fa6042e-174b-46ac-af63-a59dc462776f\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") "
Oct 31 00:16:13.400244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.400173    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\") pod \"9fa6042e-174b-46ac-af63-a59dc462776f\" (UID: \"9fa6042e-174b-46ac-af63-a59dc462776f\") "
Oct 31 00:16:13.400582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.400554    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca" pod "9fa6042e-174b-46ac-af63-a59dc462776f" (UID: "9fa6042e-174b-46ac-af63-a59dc462776f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:13.400851 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.400830    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:13.406258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.406219    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb" pod "9fa6042e-174b-46ac-af63-a59dc462776f" (UID: "9fa6042e-174b-46ac-af63-a59dc462776f"). InnerVolumeSpecName "kube-api-access-qscbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:13.406481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.406451    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca" pod "9fa6042e-174b-46ac-af63-a59dc462776f" (UID: "9fa6042e-174b-46ac-af63-a59dc462776f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:16:13.502826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.502784    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qscbb\" (UniqueName: \"kubernetes.io/projected/9fa6042e-174b-46ac-af63-a59dc462776f-kube-api-access-qscbb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:13.502826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:13.502824    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa6042e-174b-46ac-af63-a59dc462776f-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:14.107281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:14.107192    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"
Oct 31 00:16:14.107448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:14.107191    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k" event={"ID":"9fa6042e-174b-46ac-af63-a59dc462776f","Type":"ContainerDied","Data":"aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8"}
Oct 31 00:16:14.107448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:14.107329    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefcfaed1a5595136e9d35d2036dd57b2efbc8f24c58caea69d209e3cc46b9b8"
Oct 31 00:16:16.971595 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:16:16.971554    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:16:19.357017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.356970    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:16:19.357712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.357695    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9fa6042e-174b-46ac-af63-a59dc462776f" containerName="revoke-keys"
Oct 31 00:16:19.357761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.357713    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa6042e-174b-46ac-af63-a59dc462776f" containerName="revoke-keys"
Oct 31 00:16:19.357867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.357857    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa6042e-174b-46ac-af63-a59dc462776f" containerName="revoke-keys"
Oct 31 00:16:19.361910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.361886    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.488801 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.488761    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.488801 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.488809    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.488833    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.488935    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.488959    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.489142    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.489205    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.489233    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmxk\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.489428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.489279    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.520106 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.520018    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a251fdfb-openshift-default-dockercfg-k8cql\""
Oct 31 00:16:19.561772 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.561729    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:16:19.591040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.590994    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591059    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591209    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591265    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591300    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmxk\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591349    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591408    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591442    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591919 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591469    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591919 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591834    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.591919 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591875    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.592155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.591951    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.592389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.592359    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.592712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.592686    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.594518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.594483    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.594759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.594733    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.655284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.655200    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.655521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.655495    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmxk\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:19.677761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:19.677341    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:20.200370 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:20.200338    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:16:20.202261 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:20.202231    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fa903e_7b55_45c8_abde_0f613c8a7d77.slice/crio-0237d9e0aa2cb91310bcc803470be450d5f195031046ca819c04c55878aba01d WatchSource:0}: Error finding container 0237d9e0aa2cb91310bcc803470be450d5f195031046ca819c04c55878aba01d: Status 404 returned error can't find the container with id 0237d9e0aa2cb91310bcc803470be450d5f195031046ca819c04c55878aba01d
Oct 31 00:16:20.205874 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:20.205836    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:20.205945 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:20.205914    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:20.205986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:20.205954    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:21.160504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.160458    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" event={"ID":"f0fa903e-7b55-45c8-abde-0f613c8a7d77","Type":"ContainerStarted","Data":"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"}
Oct 31 00:16:21.160933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.160525    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" event={"ID":"f0fa903e-7b55-45c8-abde-0f613c8a7d77","Type":"ContainerStarted","Data":"0237d9e0aa2cb91310bcc803470be450d5f195031046ca819c04c55878aba01d"}
Oct 31 00:16:21.370602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.370544    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podStartSLOduration=2.370524574 podStartE2EDuration="2.370524574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:21.350009159 +0000 UTC m=+2033.696993057" watchObservedRunningTime="2026-10-31 00:16:21.370524574 +0000 UTC m=+2033.717508525"
Oct 31 00:16:21.678211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.678170    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:21.680398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.680357    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:21.680607 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:21.680440    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:22.678418 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:22.678385    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:22.678888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:22.678693    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:23.678010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:23.677965    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:23.678230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:23.678071    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:24.678019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:24.677983    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:24.678469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:24.678055    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:25.678258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:25.678216    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:25.678671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:25.678290    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:26.678710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:26.678668    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:16:26.679171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:26.678727    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 00:16:27.684161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:27.684120    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:27.684576 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:27.684427    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:27.685440 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:27.685420    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:16:34.532163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.532122    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:16:34.538048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.537993    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.623020 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.622973    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:16:34.669499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669441    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669566    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669671    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669713    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669810    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527kb\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669900 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669865    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.669962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669945    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.670005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.669989    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.670214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.670190    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771222    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771316    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771351    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771424    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771465    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771516    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527kb\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771569    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771651    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771686    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771691    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.771861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771764    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.772053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.771992    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.772337 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.772311    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.772612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.772591    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.774435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.774409    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.774636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.774613    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.832596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.832512    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.832747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.832703    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527kb\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:34.853119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:34.853075    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:35.286818 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:35.286782    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:16:35.289163 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:35.289133    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7338b82b_a26e_4f3d_8ccf_c283e7218542.slice/crio-56002733c0878139be540e628efbb33c04b0a5d6406bb82458717b0ed9fc4edc WatchSource:0}: Error finding container 56002733c0878139be540e628efbb33c04b0a5d6406bb82458717b0ed9fc4edc: Status 404 returned error can't find the container with id 56002733c0878139be540e628efbb33c04b0a5d6406bb82458717b0ed9fc4edc
Oct 31 00:16:35.292343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:35.292301    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:35.292434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:35.292384    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:35.292434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:35.292413    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:36.313685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.313639    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" event={"ID":"7338b82b-a26e-4f3d-8ccf-c283e7218542","Type":"ContainerStarted","Data":"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"}
Oct 31 00:16:36.314135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.313716    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" event={"ID":"7338b82b-a26e-4f3d-8ccf-c283e7218542","Type":"ContainerStarted","Data":"56002733c0878139be540e628efbb33c04b0a5d6406bb82458717b0ed9fc4edc"}
Oct 31 00:16:36.525202 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.525134    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podStartSLOduration=2.525114318 podStartE2EDuration="2.525114318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:36.475366558 +0000 UTC m=+2048.822350455" watchObservedRunningTime="2026-10-31 00:16:36.525114318 +0000 UTC m=+2048.872098216"
Oct 31 00:16:36.853972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.853929    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:36.855977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.855946    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:36.856144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:36.856010    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:37.854676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:37.854622    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:37.855217 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:37.854697    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:38.857687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:38.857645    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:38.858393 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:38.858359    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:39.854517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:39.854475    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:39.854726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:39.854534    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:40.808167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:40.808121    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"]
Oct 31 00:16:40.839505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:40.839464    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-71b76a-xtcfj"]
Oct 31 00:16:40.854263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:40.854207    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:40.854444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:40.854283    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:41.830899 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:41.830863    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab35163-5c3a-4863-b512-064fe64c5694" path="/var/lib/kubelet/pods/4ab35163-5c3a-4863-b512-064fe64c5694/volumes"
Oct 31 00:16:41.854242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:41.854212    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:16:41.854396 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:41.854265    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Oct 31 00:16:42.859491 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:42.859454    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:42.859973 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:42.859843    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:42.860638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:42.860612    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:16:46.975278 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:16:46.975240    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:16:48.592481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.592441    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"]
Oct 31 00:16:48.597301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.597270    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.646452 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.646412    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.646793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.646759    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.646950 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.646933    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggzx\" (UniqueName: \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.648958 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.648931    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:16:48.695746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.695709    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:16:48.700061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.700001    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"]
Oct 31 00:16:48.748675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.748622    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggzx\" (UniqueName: \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.748844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.748688    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.748943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.748925    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.749614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.749586    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.751329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.751306    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.807200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.807167    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggzx\" (UniqueName: \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\") pod \"maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:48.915219 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:48.915133    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:49.294401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.294367    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"]
Oct 31 00:16:49.296973 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:49.296938    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebd360d_f75f_45be_9019_8e1cce769a77.slice/crio-340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a WatchSource:0}: Error finding container 340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a: Status 404 returned error can't find the container with id 340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a
Oct 31 00:16:49.415546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.415507    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct" event={"ID":"5ebd360d-f75f-45be-9019-8e1cce769a77","Type":"ContainerStarted","Data":"340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a"}
Oct 31 00:16:49.462567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.462530    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"]
Oct 31 00:16:49.468758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.468728    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.524390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.524352    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"]
Oct 31 00:16:49.560318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.560211    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhp9\" (UniqueName: \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.560318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.560290    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.560540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.560461    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.590195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.590156    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:16:49.596303 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.596276    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.655218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.655172    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:16:49.662472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662438    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhp9\" (UniqueName: \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.662658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662507    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.662658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662540    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.662658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662603    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.662829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662729    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.662881 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662836    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vh9\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.662953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.662935    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663106 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663084    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663114    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663267 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663251    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663332 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663320    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663369    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.663497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.663475    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.665557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.665531    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.721152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.721118    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhp9\" (UniqueName: \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\") pod \"maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.765711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.765650    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.765904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.765748    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.765904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.765826    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vh9\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.765904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.765897    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766050    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766083    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766200    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766231    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766353    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766432 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766382    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.766887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.766865    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.767221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.767198    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.767314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.767244    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.767314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.767282    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.769045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.769013    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.769499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.769481    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.784543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.784497    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:49.825732 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.825646    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.825732 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.825695    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vh9\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\") pod \"e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") " pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:49.912235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:49.912193    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:50.164314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.164279    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"]
Oct 31 00:16:50.165992 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:50.165961    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20938b5a_195c_4d49_9b45_0b4b19d890f3.slice/crio-0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2 WatchSource:0}: Error finding container 0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2: Status 404 returned error can't find the container with id 0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2
Oct 31 00:16:50.295809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.295774    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:16:50.353194 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:16:50.353162    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac19bf16_b851_4d0c_bb65_ef442299f0b2.slice/crio-cf11528ce10a42d396b1890f71386da6f03eac374b92b7d72cf79b32291a5ab5 WatchSource:0}: Error finding container cf11528ce10a42d396b1890f71386da6f03eac374b92b7d72cf79b32291a5ab5: Status 404 returned error can't find the container with id cf11528ce10a42d396b1890f71386da6f03eac374b92b7d72cf79b32291a5ab5
Oct 31 00:16:50.356252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.356214    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:50.356359 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.356288    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:50.356359 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.356320    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:16:50.424066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.424010    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8" event={"ID":"20938b5a-195c-4d49-9b45-0b4b19d890f3","Type":"ContainerStarted","Data":"0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2"}
Oct 31 00:16:50.426075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.426050    6033 generic.go:356] "Generic (PLEG): container finished" podID="5ebd360d-f75f-45be-9019-8e1cce769a77" containerID="aa60b25cf498d8a4573f1b132d6c61da9957afef543fe54d7aff699977c0d56f" exitCode=0
Oct 31 00:16:50.426201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.426141    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct" event={"ID":"5ebd360d-f75f-45be-9019-8e1cce769a77","Type":"ContainerDied","Data":"aa60b25cf498d8a4573f1b132d6c61da9957afef543fe54d7aff699977c0d56f"}
Oct 31 00:16:50.427857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:50.427831    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" event={"ID":"ac19bf16-b851-4d0c-bb65-ef442299f0b2","Type":"ContainerStarted","Data":"cf11528ce10a42d396b1890f71386da6f03eac374b92b7d72cf79b32291a5ab5"}
Oct 31 00:16:51.438989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.438952    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" event={"ID":"ac19bf16-b851-4d0c-bb65-ef442299f0b2","Type":"ContainerStarted","Data":"8c5a5d5529deac486dc6aabca1219fe46beb513921a66c8492696be475c6444a"}
Oct 31 00:16:51.440902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.440864    6033 generic.go:356] "Generic (PLEG): container finished" podID="20938b5a-195c-4d49-9b45-0b4b19d890f3" containerID="68168032c07dc9cd667c2bfd45638c93ea4d37b4cec56ec0754ec0e9c6bb8506" exitCode=0
Oct 31 00:16:51.441063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.440947    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8" event={"ID":"20938b5a-195c-4d49-9b45-0b4b19d890f3","Type":"ContainerDied","Data":"68168032c07dc9cd667c2bfd45638c93ea4d37b4cec56ec0754ec0e9c6bb8506"}
Oct 31 00:16:51.618621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.618508    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podStartSLOduration=2.618492118 podStartE2EDuration="2.618492118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:51.552803771 +0000 UTC m=+2063.899787671" watchObservedRunningTime="2026-10-31 00:16:51.618492118 +0000 UTC m=+2063.965476015"
Oct 31 00:16:51.656274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.656237    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:51.819946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.819896    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\") pod \"5ebd360d-f75f-45be-9019-8e1cce769a77\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") "
Oct 31 00:16:51.820175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.820055    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\") pod \"5ebd360d-f75f-45be-9019-8e1cce769a77\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") "
Oct 31 00:16:51.820402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.820320    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\" (UniqueName: \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\") pod \"5ebd360d-f75f-45be-9019-8e1cce769a77\" (UID: \"5ebd360d-f75f-45be-9019-8e1cce769a77\") "
Oct 31 00:16:51.820402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.820351    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca" pod "5ebd360d-f75f-45be-9019-8e1cce769a77" (UID: "5ebd360d-f75f-45be-9019-8e1cce769a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:51.820993 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.820965    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:51.823938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.823105    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx" pod "5ebd360d-f75f-45be-9019-8e1cce769a77" (UID: "5ebd360d-f75f-45be-9019-8e1cce769a77"). InnerVolumeSpecName "kube-api-access-sggzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:51.823938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.823880    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca" pod "5ebd360d-f75f-45be-9019-8e1cce769a77" (UID: "5ebd360d-f75f-45be-9019-8e1cce769a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:16:51.912464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.912368    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:51.914100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.914071    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 00:16:51.914249 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.914124    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 00:16:51.923085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.923012    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggzx\" (UniqueName: \"kubernetes.io/projected/5ebd360d-f75f-45be-9019-8e1cce769a77-kube-api-access-sggzx\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:51.923280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:51.923256    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ebd360d-f75f-45be-9019-8e1cce769a77-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:52.450408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.450372    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"
Oct 31 00:16:52.450846 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.450395    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct" event={"ID":"5ebd360d-f75f-45be-9019-8e1cce769a77","Type":"ContainerDied","Data":"340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a"}
Oct 31 00:16:52.450846 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.450454    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340bebc461672750baf709fda627ea6f7df60e2bda9655d9cca72f64b0d2c08a"
Oct 31 00:16:52.664412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.664374    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:52.835988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.835942    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\" (UniqueName: \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\") pod \"20938b5a-195c-4d49-9b45-0b4b19d890f3\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") "
Oct 31 00:16:52.836197 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.836179    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\") pod \"20938b5a-195c-4d49-9b45-0b4b19d890f3\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") "
Oct 31 00:16:52.836463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.836441    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\") pod \"20938b5a-195c-4d49-9b45-0b4b19d890f3\" (UID: \"20938b5a-195c-4d49-9b45-0b4b19d890f3\") "
Oct 31 00:16:52.836598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.836573    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca" pod "20938b5a-195c-4d49-9b45-0b4b19d890f3" (UID: "20938b5a-195c-4d49-9b45-0b4b19d890f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:16:52.837105 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.837079    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:52.838293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.838265    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9" pod "20938b5a-195c-4d49-9b45-0b4b19d890f3" (UID: "20938b5a-195c-4d49-9b45-0b4b19d890f3"). InnerVolumeSpecName "kube-api-access-mxhp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:16:52.838621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.838600    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca" pod "20938b5a-195c-4d49-9b45-0b4b19d890f3" (UID: "20938b5a-195c-4d49-9b45-0b4b19d890f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:16:52.913348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.913308    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 00:16:52.913539 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.913365    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 00:16:52.938675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.938576    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20938b5a-195c-4d49-9b45-0b4b19d890f3-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:52.938675 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:52.938633    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhp9\" (UniqueName: \"kubernetes.io/projected/20938b5a-195c-4d49-9b45-0b4b19d890f3-kube-api-access-mxhp9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:16:53.460685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:53.460646    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"
Oct 31 00:16:53.460685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:53.460674    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8" event={"ID":"20938b5a-195c-4d49-9b45-0b4b19d890f3","Type":"ContainerDied","Data":"0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2"}
Oct 31 00:16:53.461152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:53.460725    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6da955017aa27ed592b05a9ca509e97dbca752ab9d337e3ff10a81c11feca2"
Oct 31 00:16:53.913649 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:53.913615    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 00:16:53.913840 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:53.913675    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 00:16:54.913485 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:54.913438    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 00:16:54.913998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:54.913511    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 00:16:55.913665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:55.913621    6033 patch_prober.go:29] interesting pod/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 00:16:55.914215 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:55.913684    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 00:16:57.029184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:57.029129    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:57.029985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:57.029953    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:16:57.031286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:16:57.031260    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:17:01.559454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.559412    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"]
Oct 31 00:17:01.561139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561101    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5ebd360d-f75f-45be-9019-8e1cce769a77" containerName="revoke-keys"
Oct 31 00:17:01.561139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561130    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ebd360d-f75f-45be-9019-8e1cce769a77" containerName="revoke-keys"
Oct 31 00:17:01.561367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561154    6033 cpu_manager.go:414] "containerMap: removing container" podUID="20938b5a-195c-4d49-9b45-0b4b19d890f3" containerName="revoke-keys"
Oct 31 00:17:01.561367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561162    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20938b5a-195c-4d49-9b45-0b4b19d890f3" containerName="revoke-keys"
Oct 31 00:17:01.561514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561436    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ebd360d-f75f-45be-9019-8e1cce769a77" containerName="revoke-keys"
Oct 31 00:17:01.561580 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.561521    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="20938b5a-195c-4d49-9b45-0b4b19d890f3" containerName="revoke-keys"
Oct 31 00:17:01.566229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.566197    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.598425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.598380    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.598611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.598591    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxk2\" (UniqueName: \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.598730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.598702    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.623407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.623368    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"]
Oct 31 00:17:01.665662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.665625    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:01.665880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.665680    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:17:01.721686 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.721639    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.721880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.721835    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxk2\" (UniqueName: \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.721934 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.721915    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.722893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.722863    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.735089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.725866    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.788322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.782105    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxk2\" (UniqueName: \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\") pod \"maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:01.883353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:01.883250    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:02.266413 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:17:02.266381    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a64369_d4f7_400e_adb0_c6b6af995559.slice/crio-5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365 WatchSource:0}: Error finding container 5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365: Status 404 returned error can't find the container with id 5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365
Oct 31 00:17:02.268460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:02.268368    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"]
Oct 31 00:17:02.531285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:02.531193    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz" event={"ID":"a4a64369-d4f7-400e-adb0-c6b6af995559","Type":"ContainerStarted","Data":"5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365"}
Oct 31 00:17:03.541973 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:03.541939    6033 generic.go:356] "Generic (PLEG): container finished" podID="a4a64369-d4f7-400e-adb0-c6b6af995559" containerID="a2d928b462319f9582e79a7c5b97f1a12cc4294271207def8e339475077479aa" exitCode=0
Oct 31 00:17:03.542390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:03.542023    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz" event={"ID":"a4a64369-d4f7-400e-adb0-c6b6af995559","Type":"ContainerDied","Data":"a2d928b462319f9582e79a7c5b97f1a12cc4294271207def8e339475077479aa"}
Oct 31 00:17:04.718718 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.718687    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:04.769251 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.769219    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\") pod \"a4a64369-d4f7-400e-adb0-c6b6af995559\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") "
Oct 31 00:17:04.769451 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.769319    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\") pod \"a4a64369-d4f7-400e-adb0-c6b6af995559\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") "
Oct 31 00:17:04.769523 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.769455    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\" (UniqueName: \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\") pod \"a4a64369-d4f7-400e-adb0-c6b6af995559\" (UID: \"a4a64369-d4f7-400e-adb0-c6b6af995559\") "
Oct 31 00:17:04.769678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.769648    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca" pod "a4a64369-d4f7-400e-adb0-c6b6af995559" (UID: "a4a64369-d4f7-400e-adb0-c6b6af995559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:04.770123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.770090    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:04.771610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.771580    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca" pod "a4a64369-d4f7-400e-adb0-c6b6af995559" (UID: "a4a64369-d4f7-400e-adb0-c6b6af995559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:04.771610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.771594    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2" pod "a4a64369-d4f7-400e-adb0-c6b6af995559" (UID: "a4a64369-d4f7-400e-adb0-c6b6af995559"). InnerVolumeSpecName "kube-api-access-7jxk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:04.871599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.871509    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a64369-d4f7-400e-adb0-c6b6af995559-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:04.871599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.871548    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxk2\" (UniqueName: \"kubernetes.io/projected/a4a64369-d4f7-400e-adb0-c6b6af995559-kube-api-access-7jxk2\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:04.879165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.879127    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:17:04.879468 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.879416    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" podUID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" containerName="maas-api" containerID="cri-o://41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc" gracePeriod=30
Oct 31 00:17:04.884426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.882928    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"]
Oct 31 00:17:04.884426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.883932    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a4a64369-d4f7-400e-adb0-c6b6af995559" containerName="revoke-keys"
Oct 31 00:17:04.884426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.883953    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a64369-d4f7-400e-adb0-c6b6af995559" containerName="revoke-keys"
Oct 31 00:17:04.884426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.884172    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a64369-d4f7-400e-adb0-c6b6af995559" containerName="revoke-keys"
Oct 31 00:17:04.889447 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.889323    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:04.965125 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.965078    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"]
Oct 31 00:17:04.973105 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.973073    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:04.973306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.973146    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:04.973436 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:04.973404    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsvx\" (UniqueName: \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.075763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.074851    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsvx\" (UniqueName: \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.075763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.075003    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.075763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.075097    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.077046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.076992    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.272113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.222968    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.295724 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.295651    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsvx\" (UniqueName: \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\") pod \"maas-api-revoke-keys-e2e-inf-a-03c703-4v79q\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.424284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.423170    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:17:05.554116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.545645    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\") pod \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") "
Oct 31 00:17:05.554116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.545720    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\") pod \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") "
Oct 31 00:17:05.554116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.545772    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\" (UniqueName: \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\") pod \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") "
Oct 31 00:17:05.554116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.546106    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") pod \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\" (UID: \"364c444f-bc4e-4d7c-8112-00e0fdf56bcb\") "
Oct 31 00:17:05.566108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.561199    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs" pod "364c444f-bc4e-4d7c-8112-00e0fdf56bcb" (UID: "364c444f-bc4e-4d7c-8112-00e0fdf56bcb"). InnerVolumeSpecName "kube-api-access-x8shs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:05.566108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.562138    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls" pod "364c444f-bc4e-4d7c-8112-00e0fdf56bcb" (UID: "364c444f-bc4e-4d7c-8112-00e0fdf56bcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:05.566108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.565914    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:05.566692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.566626    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca" pod "364c444f-bc4e-4d7c-8112-00e0fdf56bcb" (UID: "364c444f-bc4e-4d7c-8112-00e0fdf56bcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:05.588779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.587789    6033 generic.go:356] "Generic (PLEG): container finished" podID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" containerID="41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc" exitCode=0
Oct 31 00:17:05.588779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.587968    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" event={"ID":"364c444f-bc4e-4d7c-8112-00e0fdf56bcb","Type":"ContainerDied","Data":"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"}
Oct 31 00:17:05.588779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.588066    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw" event={"ID":"364c444f-bc4e-4d7c-8112-00e0fdf56bcb","Type":"ContainerDied","Data":"9fec64150ab46179177f3339b4f2f99e8aaef90305b970d7c1d9f947c50ff827"}
Oct 31 00:17:05.588779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.588104    6033 scope.go:122] "RemoveContainer" containerID="41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"
Oct 31 00:17:05.588779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.588268    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"
Oct 31 00:17:05.592805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.592768    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz" event={"ID":"a4a64369-d4f7-400e-adb0-c6b6af995559","Type":"ContainerDied","Data":"5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365"}
Oct 31 00:17:05.593134 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.593114    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5392902136f5224214ab9b354e9b8ca5e60ae64fb6238cf3d95565f0439bc365"
Oct 31 00:17:05.593377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.593343    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"
Oct 31 00:17:05.597882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.597844    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs" pod "364c444f-bc4e-4d7c-8112-00e0fdf56bcb" (UID: "364c444f-bc4e-4d7c-8112-00e0fdf56bcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:05.624248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.620652    6033 scope.go:122] "RemoveContainer" containerID="41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"
Oct 31 00:17:05.624248 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:05.621144    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc\": container with ID starting with 41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc not found: ID does not exist" containerID="41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"
Oct 31 00:17:05.624248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.621189    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc"} err="failed to get container status \"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc\": rpc error: code = NotFound desc = could not find container \"41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc\": container with ID starting with 41b469e0d86b2cdf47fc1de6eeccc643cd03c96b4f23a02af28b6bc6e578e2dc not found: ID does not exist"
Oct 31 00:17:05.653640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.649415    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:05.653640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.649457    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:05.653640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.649482    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8shs\" (UniqueName: \"kubernetes.io/projected/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-kube-api-access-x8shs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:05.653640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.649499    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364c444f-bc4e-4d7c-8112-00e0fdf56bcb-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:05.999246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:05.998673    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:17:06.007620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:06.007552    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-348a33-586dcb694-bldzw"]
Oct 31 00:17:06.030529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:06.030491    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"]
Oct 31 00:17:06.047053 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:17:06.046990    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae08413_7163_4610_b00a_7746990ec658.slice/crio-5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917 WatchSource:0}: Error finding container 5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917: Status 404 returned error can't find the container with id 5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917
Oct 31 00:17:06.603313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:06.603268    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q" event={"ID":"4ae08413-7163-4610-b00a-7746990ec658","Type":"ContainerStarted","Data":"46d4ae82b16458aad78458adf93942ae80013651c8d225ee57a913f0cbfd6f98"}
Oct 31 00:17:06.603313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:06.603321    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q" event={"ID":"4ae08413-7163-4610-b00a-7746990ec658","Type":"ContainerStarted","Data":"5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917"}
Oct 31 00:17:06.728099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:06.727972    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q" podStartSLOduration=2.727957584 podStartE2EDuration="2.727957584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:06.72553303 +0000 UTC m=+2079.072516927" watchObservedRunningTime="2026-10-31 00:17:06.727957584 +0000 UTC m=+2079.074941460"
Oct 31 00:17:07.164474 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.164437    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"]
Oct 31 00:17:07.168253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.168221    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-348a33-hbvt8"]
Oct 31 00:17:07.612778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.612742    6033 generic.go:356] "Generic (PLEG): container finished" podID="4ae08413-7163-4610-b00a-7746990ec658" containerID="46d4ae82b16458aad78458adf93942ae80013651c8d225ee57a913f0cbfd6f98" exitCode=0
Oct 31 00:17:07.612997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.612800    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q" event={"ID":"4ae08413-7163-4610-b00a-7746990ec658","Type":"ContainerDied","Data":"46d4ae82b16458aad78458adf93942ae80013651c8d225ee57a913f0cbfd6f98"}
Oct 31 00:17:07.831791 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.831754    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20938b5a-195c-4d49-9b45-0b4b19d890f3" path="/var/lib/kubelet/pods/20938b5a-195c-4d49-9b45-0b4b19d890f3/volumes"
Oct 31 00:17:07.832294 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:07.832278    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" path="/var/lib/kubelet/pods/364c444f-bc4e-4d7c-8112-00e0fdf56bcb/volumes"
Oct 31 00:17:08.823044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:08.823008    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:09.002895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.002859    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:17:09.003247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.003087    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\") pod \"4ae08413-7163-4610-b00a-7746990ec658\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") "
Oct 31 00:17:09.003247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.003188    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\" (UniqueName: \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\") pod \"4ae08413-7163-4610-b00a-7746990ec658\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") "
Oct 31 00:17:09.003469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.003447    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\") pod \"4ae08413-7163-4610-b00a-7746990ec658\" (UID: \"4ae08413-7163-4610-b00a-7746990ec658\") "
Oct 31 00:17:09.003963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.003927    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca" pod "4ae08413-7163-4610-b00a-7746990ec658" (UID: "4ae08413-7163-4610-b00a-7746990ec658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:09.004285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.004265    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae08413-7163-4610-b00a-7746990ec658-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:09.005903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.005874    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx" pod "4ae08413-7163-4610-b00a-7746990ec658" (UID: "4ae08413-7163-4610-b00a-7746990ec658"). InnerVolumeSpecName "kube-api-access-cnsvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:09.006006 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.005991    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca" pod "4ae08413-7163-4610-b00a-7746990ec658" (UID: "4ae08413-7163-4610-b00a-7746990ec658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:09.105542 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.105507    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae08413-7163-4610-b00a-7746990ec658-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:09.105542 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.105535    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnsvx\" (UniqueName: \"kubernetes.io/projected/4ae08413-7163-4610-b00a-7746990ec658-kube-api-access-cnsvx\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:09.652055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.651281    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q" event={"ID":"4ae08413-7163-4610-b00a-7746990ec658","Type":"ContainerDied","Data":"5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917"}
Oct 31 00:17:09.652055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.651348    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dbb63712bbe6e328e655fabcbae038c8c7b7dffcd42b0ba8757ab6757301917"
Oct 31 00:17:09.652055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:09.651433    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"
Oct 31 00:17:12.559930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:12.559889    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:17:16.986345 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:16.986300    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:17:17.565053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.562105    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" podUID="dfd217a9-2414-4b86-bc25-f8462678e92e" containerName="payload-pre-processing" containerID="cri-o://6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792" gracePeriod=25
Oct 31 00:17:17.710078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.710016    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"]
Oct 31 00:17:17.712225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.712189    6033 cpu_manager.go:414] "containerMap: removing container" podUID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" containerName="maas-api"
Oct 31 00:17:17.712459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.712440    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" containerName="maas-api"
Oct 31 00:17:17.712695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.712673    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4ae08413-7163-4610-b00a-7746990ec658" containerName="revoke-keys"
Oct 31 00:17:17.712829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.712810    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae08413-7163-4610-b00a-7746990ec658" containerName="revoke-keys"
Oct 31 00:17:17.730611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.730579    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae08413-7163-4610-b00a-7746990ec658" containerName="revoke-keys"
Oct 31 00:17:17.730807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.730780    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="364c444f-bc4e-4d7c-8112-00e0fdf56bcb" containerName="maas-api"
Oct 31 00:17:17.735450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.735420    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.774133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.774086    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.778316 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.778275    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.794062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.778765    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdf6\" (UniqueName: \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.805050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.800398    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"]
Oct 31 00:17:17.820044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.819474    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:17:17.820044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.819544    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:17.897648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.895727    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdf6\" (UniqueName: \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.897648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.895981    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.897648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.896013    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.897648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.896789    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.914696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.914581    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:17.985483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:17.985403    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdf6\" (UniqueName: \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\") pod \"maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:18.068073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.068021    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:18.150739 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.150704    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:17:18.305069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.304989    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\") pod \"dfd217a9-2414-4b86-bc25-f8462678e92e\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") "
Oct 31 00:17:18.305267 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.305174    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\" (UniqueName: \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\") pod \"dfd217a9-2414-4b86-bc25-f8462678e92e\" (UID: \"dfd217a9-2414-4b86-bc25-f8462678e92e\") "
Oct 31 00:17:18.307658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.307616    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8" pod "dfd217a9-2414-4b86-bc25-f8462678e92e" (UID: "dfd217a9-2414-4b86-bc25-f8462678e92e"). InnerVolumeSpecName "kube-api-access-r7nd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:18.310280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.310249    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume" pod "dfd217a9-2414-4b86-bc25-f8462678e92e" (UID: "dfd217a9-2414-4b86-bc25-f8462678e92e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:18.407861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.407757    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd217a9-2414-4b86-bc25-f8462678e92e-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.407861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.407799    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nd8\" (UniqueName: \"kubernetes.io/projected/dfd217a9-2414-4b86-bc25-f8462678e92e-kube-api-access-r7nd8\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:18.501052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.500990    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"]
Oct 31 00:17:18.502609 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:17:18.502575    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd836ce_5cda_403c_9ad5_174019f6d5b2.slice/crio-9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3 WatchSource:0}: Error finding container 9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3: Status 404 returned error can't find the container with id 9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3
Oct 31 00:17:18.740401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.740368    6033 generic.go:356] "Generic (PLEG): container finished" podID="dfd217a9-2414-4b86-bc25-f8462678e92e" containerID="6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792" exitCode=0
Oct 31 00:17:18.740592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.740415    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"
Oct 31 00:17:18.740592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.740441    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" event={"ID":"dfd217a9-2414-4b86-bc25-f8462678e92e","Type":"ContainerDied","Data":"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"}
Oct 31 00:17:18.740592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.740501    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7" event={"ID":"dfd217a9-2414-4b86-bc25-f8462678e92e","Type":"ContainerDied","Data":"3ee2bc24aa4f692cb825c2866e33ab1b51f7bb530199ee06df34da2cc2e24b27"}
Oct 31 00:17:18.740592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.740534    6033 scope.go:122] "RemoveContainer" containerID="6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"
Oct 31 00:17:18.742347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.742321    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw" event={"ID":"2bd836ce-5cda-403c-9ad5-174019f6d5b2","Type":"ContainerStarted","Data":"9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3"}
Oct 31 00:17:18.753222 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.753201    6033 scope.go:122] "RemoveContainer" containerID="6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"
Oct 31 00:17:18.753632 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:18.753611    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792\": container with ID starting with 6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792 not found: ID does not exist" containerID="6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"
Oct 31 00:17:18.753717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.753644    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792"} err="failed to get container status \"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792\": rpc error: code = NotFound desc = could not find container \"6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792\": container with ID starting with 6ddb900e0ebfc1a07e6bf0edd81f1338c34761c8f34b6ac1b15d1f6ec3814792 not found: ID does not exist"
Oct 31 00:17:18.762084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.761996    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"]
Oct 31 00:17:18.768967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.768924    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab0ac9-k5vgc"]
Oct 31 00:17:18.897273 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.897236    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:17:18.902405 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:18.902366    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a251fdfb-6f8487684c-s2hz7"]
Oct 31 00:17:19.752921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:19.752886    6033 generic.go:356] "Generic (PLEG): container finished" podID="2bd836ce-5cda-403c-9ad5-174019f6d5b2" containerID="3295d2284918c6566e06335254d8ff1aed593e29f880034ff1a8c21ed3f5696f" exitCode=0
Oct 31 00:17:19.753364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:19.752970    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw" event={"ID":"2bd836ce-5cda-403c-9ad5-174019f6d5b2","Type":"ContainerDied","Data":"3295d2284918c6566e06335254d8ff1aed593e29f880034ff1a8c21ed3f5696f"}
Oct 31 00:17:19.830343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:19.830251    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4169271-f966-4c85-98b2-ac6adc9403f6" path="/var/lib/kubelet/pods/d4169271-f966-4c85-98b2-ac6adc9403f6/volumes"
Oct 31 00:17:19.830817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:19.830793    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd217a9-2414-4b86-bc25-f8462678e92e" path="/var/lib/kubelet/pods/dfd217a9-2414-4b86-bc25-f8462678e92e/volumes"
Oct 31 00:17:20.947873 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:20.947838    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:21.051730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.051632    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\") pod \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") "
Oct 31 00:17:21.051915 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.051757    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\" (UniqueName: \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\") pod \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") "
Oct 31 00:17:21.051915 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.051832    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\") pod \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\" (UID: \"2bd836ce-5cda-403c-9ad5-174019f6d5b2\") "
Oct 31 00:17:21.052294 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.052269    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca" pod "2bd836ce-5cda-403c-9ad5-174019f6d5b2" (UID: "2bd836ce-5cda-403c-9ad5-174019f6d5b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:21.052660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.052633    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:21.053928 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.053897    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca" pod "2bd836ce-5cda-403c-9ad5-174019f6d5b2" (UID: "2bd836ce-5cda-403c-9ad5-174019f6d5b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:21.054116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.054098    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6" pod "2bd836ce-5cda-403c-9ad5-174019f6d5b2" (UID: "2bd836ce-5cda-403c-9ad5-174019f6d5b2"). InnerVolumeSpecName "kube-api-access-5sdf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:21.154483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.154443    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd836ce-5cda-403c-9ad5-174019f6d5b2-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:21.154483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.154481    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdf6\" (UniqueName: \"kubernetes.io/projected/2bd836ce-5cda-403c-9ad5-174019f6d5b2-kube-api-access-5sdf6\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:21.446472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.446355    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:17:21.466921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.466880    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 31 00:17:21.492293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.492256    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 31 00:17:21.773818 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.773774    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw" event={"ID":"2bd836ce-5cda-403c-9ad5-174019f6d5b2","Type":"ContainerDied","Data":"9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3"}
Oct 31 00:17:21.774055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.773851    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de360d2e5309837001fd42bffa6abe78929c4cd8579f343d1c9ca205c212ba3"
Oct 31 00:17:21.774055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:21.773801    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"
Oct 31 00:17:25.051992 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.051949    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"]
Oct 31 00:17:25.097931 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.097892    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e3177f-kdlnq"]
Oct 31 00:17:25.467716 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.467569    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"]
Oct 31 00:17:25.468530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468498    6033 cpu_manager.go:414] "containerMap: removing container" podUID="2bd836ce-5cda-403c-9ad5-174019f6d5b2" containerName="revoke-keys"
Oct 31 00:17:25.468530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468529    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd836ce-5cda-403c-9ad5-174019f6d5b2" containerName="revoke-keys"
Oct 31 00:17:25.468731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468565    6033 cpu_manager.go:414] "containerMap: removing container" podUID="dfd217a9-2414-4b86-bc25-f8462678e92e" containerName="payload-pre-processing"
Oct 31 00:17:25.468731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468571    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd217a9-2414-4b86-bc25-f8462678e92e" containerName="payload-pre-processing"
Oct 31 00:17:25.468731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468712    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd836ce-5cda-403c-9ad5-174019f6d5b2" containerName="revoke-keys"
Oct 31 00:17:25.468731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.468725    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd217a9-2414-4b86-bc25-f8462678e92e" containerName="payload-pre-processing"
Oct 31 00:17:25.473923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.473899    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.521171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.521124    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.521395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.521280    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.521395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.521319    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnkz\" (UniqueName: \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.523622 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.523592    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:25.523773 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.523722    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:17:25.524885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.524853    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"]
Oct 31 00:17:25.623385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.623348    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.623592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.623436    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.623592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.623462    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnkz\" (UniqueName: \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.624149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.624123    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.626184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.626155    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.690341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.690307    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnkz\" (UniqueName: \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\") pod \"maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.789569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.789535    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:25.845424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:25.845065    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c35045-b469-45f7-8b20-b31991827768" path="/var/lib/kubelet/pods/c7c35045-b469-45f7-8b20-b31991827768/volumes"
Oct 31 00:17:26.248876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.248778    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"]
Oct 31 00:17:26.252712 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:17:26.252683    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71abc5d3_394d_4917_8fd7_094367dd4e51.slice/crio-0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f WatchSource:0}: Error finding container 0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f: Status 404 returned error can't find the container with id 0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f
Oct 31 00:17:26.717405 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.717366    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"]
Oct 31 00:17:26.722389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.722357    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.760360 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.760319    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk98r\" (UniqueName: \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.760548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.760370    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.760548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.760512    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.779997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.779957    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"]
Oct 31 00:17:26.850161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.850060    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf" event={"ID":"71abc5d3-394d-4917-8fd7-094367dd4e51","Type":"ContainerStarted","Data":"28a4b5e978d211a7be64e4fba18802cdeedf11539a716fb37ef823026b6351d3"}
Oct 31 00:17:26.850161 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.850132    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf" event={"ID":"71abc5d3-394d-4917-8fd7-094367dd4e51","Type":"ContainerStarted","Data":"0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f"}
Oct 31 00:17:26.862329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.862286    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk98r\" (UniqueName: \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.862514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.862365    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.862579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.862548    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.863127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.863100    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.865100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.865080    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.920849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.920817    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk98r\" (UniqueName: \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:26.962645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:26.962585    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf" podStartSLOduration=1.9625700149999998 podStartE2EDuration="1.962570015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:26.959640286 +0000 UTC m=+2099.306624184" watchObservedRunningTime="2026-10-31 00:17:26.962570015 +0000 UTC m=+2099.309553914"
Oct 31 00:17:27.039347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.039309    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:27.419814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.419775    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"]
Oct 31 00:17:27.421021 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:17:27.420995    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc4c009_958c_43e9_a338_6af211f423e5.slice/crio-62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27 WatchSource:0}: Error finding container 62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27: Status 404 returned error can't find the container with id 62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27
Oct 31 00:17:27.859878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.859787    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t" event={"ID":"fbc4c009-958c-43e9-a338-6af211f423e5","Type":"ContainerStarted","Data":"c798f9ebcbdc1af93fae35036ecedf56b264e1648acf5eafdf0627808b464c83"}
Oct 31 00:17:27.859878 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.859864    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t" event={"ID":"fbc4c009-958c-43e9-a338-6af211f423e5","Type":"ContainerStarted","Data":"62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27"}
Oct 31 00:17:27.862810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.862781    6033 generic.go:356] "Generic (PLEG): container finished" podID="71abc5d3-394d-4917-8fd7-094367dd4e51" containerID="28a4b5e978d211a7be64e4fba18802cdeedf11539a716fb37ef823026b6351d3" exitCode=0
Oct 31 00:17:27.862963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.862861    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf" event={"ID":"71abc5d3-394d-4917-8fd7-094367dd4e51","Type":"ContainerDied","Data":"28a4b5e978d211a7be64e4fba18802cdeedf11539a716fb37ef823026b6351d3"}
Oct 31 00:17:27.971722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:27.971665    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t" podStartSLOduration=1.971647505 podStartE2EDuration="1.971647505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:27.968226286 +0000 UTC m=+2100.315210183" watchObservedRunningTime="2026-10-31 00:17:27.971647505 +0000 UTC m=+2100.318631653"
Oct 31 00:17:28.873461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:28.873421    6033 generic.go:356] "Generic (PLEG): container finished" podID="fbc4c009-958c-43e9-a338-6af211f423e5" containerID="c798f9ebcbdc1af93fae35036ecedf56b264e1648acf5eafdf0627808b464c83" exitCode=0
Oct 31 00:17:28.873948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:28.873501    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t" event={"ID":"fbc4c009-958c-43e9-a338-6af211f423e5","Type":"ContainerDied","Data":"c798f9ebcbdc1af93fae35036ecedf56b264e1648acf5eafdf0627808b464c83"}
Oct 31 00:17:29.075407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.075371    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:29.198666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.198632    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\") pod \"71abc5d3-394d-4917-8fd7-094367dd4e51\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") "
Oct 31 00:17:29.198870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.198835    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\" (UniqueName: \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\") pod \"71abc5d3-394d-4917-8fd7-094367dd4e51\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") "
Oct 31 00:17:29.198997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.198977    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\") pod \"71abc5d3-394d-4917-8fd7-094367dd4e51\" (UID: \"71abc5d3-394d-4917-8fd7-094367dd4e51\") "
Oct 31 00:17:29.199131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.199109    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca" pod "71abc5d3-394d-4917-8fd7-094367dd4e51" (UID: "71abc5d3-394d-4917-8fd7-094367dd4e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:29.199743 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.199713    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:29.201371 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.201177    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz" pod "71abc5d3-394d-4917-8fd7-094367dd4e51" (UID: "71abc5d3-394d-4917-8fd7-094367dd4e51"). InnerVolumeSpecName "kube-api-access-bvnkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:29.201517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.201395    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca" pod "71abc5d3-394d-4917-8fd7-094367dd4e51" (UID: "71abc5d3-394d-4917-8fd7-094367dd4e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:29.301651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.301559    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71abc5d3-394d-4917-8fd7-094367dd4e51-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:29.301651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.301599    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvnkz\" (UniqueName: \"kubernetes.io/projected/71abc5d3-394d-4917-8fd7-094367dd4e51-kube-api-access-bvnkz\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:29.882127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.882096    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"
Oct 31 00:17:29.882594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.882099    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf" event={"ID":"71abc5d3-394d-4917-8fd7-094367dd4e51","Type":"ContainerDied","Data":"0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f"}
Oct 31 00:17:29.882594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:29.882222    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ced22f9ce8b1cd9b4dcb1208ac4dfc024f5a726fefa8d811c1ae1b1f393ae6f"
Oct 31 00:17:30.070833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.070801    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:30.233115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.231156    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\") pod \"fbc4c009-958c-43e9-a338-6af211f423e5\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") "
Oct 31 00:17:30.233115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.231243    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\") pod \"fbc4c009-958c-43e9-a338-6af211f423e5\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") "
Oct 31 00:17:30.233115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.231323    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\" (UniqueName: \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\") pod \"fbc4c009-958c-43e9-a338-6af211f423e5\" (UID: \"fbc4c009-958c-43e9-a338-6af211f423e5\") "
Oct 31 00:17:30.233115 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.232810    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca" pod "fbc4c009-958c-43e9-a338-6af211f423e5" (UID: "fbc4c009-958c-43e9-a338-6af211f423e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:30.238621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.235931    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca" pod "fbc4c009-958c-43e9-a338-6af211f423e5" (UID: "fbc4c009-958c-43e9-a338-6af211f423e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:30.245623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.245536    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r" pod "fbc4c009-958c-43e9-a338-6af211f423e5" (UID: "fbc4c009-958c-43e9-a338-6af211f423e5"). InnerVolumeSpecName "kube-api-access-wk98r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:30.333538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.333502    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:30.333538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.333534    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc4c009-958c-43e9-a338-6af211f423e5-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:30.333538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.333544    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk98r\" (UniqueName: \"kubernetes.io/projected/fbc4c009-958c-43e9-a338-6af211f423e5-kube-api-access-wk98r\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:30.892549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.892511    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t" event={"ID":"fbc4c009-958c-43e9-a338-6af211f423e5","Type":"ContainerDied","Data":"62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27"}
Oct 31 00:17:30.892549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.892538    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"
Oct 31 00:17:30.892979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:30.892566    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ae2346cb4bd6d9fe6dbc0b49cf6c6419a692b9a14f901abad922080327de27"
Oct 31 00:17:36.469598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.469535    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main" containerID="cri-o://4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9" gracePeriod=45
Oct 31 00:17:36.516078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.516003    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main" containerID="cri-o://bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005" gracePeriod=45
Oct 31 00:17:36.802726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.802668    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main" containerID="cri-o://1e2921034f5f34810e64757ac698b3fd748019a3999116cb75a508ad8093d500" gracePeriod=45
Oct 31 00:17:36.810258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.810226    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:17:36.892905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.892878    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:17:36.932616 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932582    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\" (UniqueName: \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.932814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932670    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.932814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932750    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.932814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932787    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.933004 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932820    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.933004 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.932889    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\") pod \"52aaf67b-7519-484b-af54-a512508d32ed\" (UID: \"52aaf67b-7519-484b-af54-a512508d32ed\") "
Oct 31 00:17:36.933198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933107    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:36.933367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933336    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:36.933367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933348    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:36.933697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933677    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.933776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933703    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.933776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.933721    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.935285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.935251    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "kube-api-access-wkgws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:36.935285 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.935274    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:36.935461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.935283    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs" pod "52aaf67b-7519-484b-af54-a512508d32ed" (UID: "52aaf67b-7519-484b-af54-a512508d32ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:36.938817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.938789    6033 generic.go:356] "Generic (PLEG): container finished" podID="52aaf67b-7519-484b-af54-a512508d32ed" containerID="4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9" exitCode=0
Oct 31 00:17:36.938951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.938860    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"
Oct 31 00:17:36.938951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.938870    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" event={"ID":"52aaf67b-7519-484b-af54-a512508d32ed","Type":"ContainerDied","Data":"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"}
Oct 31 00:17:36.938951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.938913    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4" event={"ID":"52aaf67b-7519-484b-af54-a512508d32ed","Type":"ContainerDied","Data":"a65f285f5ce098f056b7c9e365b82b38ae26f106b38f7ab7a24f067c630ac181"}
Oct 31 00:17:36.938951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.938934    6033 scope.go:122] "RemoveContainer" containerID="4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"
Oct 31 00:17:36.941346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.941087    6033 generic.go:356] "Generic (PLEG): container finished" podID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerID="1e2921034f5f34810e64757ac698b3fd748019a3999116cb75a508ad8093d500" exitCode=0
Oct 31 00:17:36.941346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.941182    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" event={"ID":"c5d19655-4df5-42bb-aca8-d782c33fa3df","Type":"ContainerDied","Data":"1e2921034f5f34810e64757ac698b3fd748019a3999116cb75a508ad8093d500"}
Oct 31 00:17:36.943221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.943201    6033 generic.go:356] "Generic (PLEG): container finished" podID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerID="bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005" exitCode=0
Oct 31 00:17:36.943348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.943246    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"
Oct 31 00:17:36.943348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.943249    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" event={"ID":"54ed9707-5cc1-476c-b4f6-1659f402d75b","Type":"ContainerDied","Data":"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"}
Oct 31 00:17:36.943348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.943275    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd" event={"ID":"54ed9707-5cc1-476c-b4f6-1659f402d75b","Type":"ContainerDied","Data":"0c924a352b8d295c5d55b5aef1c1baa521ccbd533b55ad50dac432e511f2923b"}
Oct 31 00:17:36.955501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.955478    6033 scope.go:122] "RemoveContainer" containerID="4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"
Oct 31 00:17:36.955897 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:36.955867    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9\": container with ID starting with 4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9 not found: ID does not exist" containerID="4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"
Oct 31 00:17:36.955988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.955908    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9"} err="failed to get container status \"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9\": rpc error: code = NotFound desc = could not find container \"4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9\": container with ID starting with 4d90cda7505cca6455b41e4bbe0fcd68c962adc5165985985943cc0768f3a1a9 not found: ID does not exist"
Oct 31 00:17:36.955988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.955937    6033 scope.go:122] "RemoveContainer" containerID="bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"
Oct 31 00:17:36.992309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.977909    6033 scope.go:122] "RemoveContainer" containerID="bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"
Oct 31 00:17:36.992309 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:36.978882    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005\": container with ID starting with bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005 not found: ID does not exist" containerID="bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"
Oct 31 00:17:36.992309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:36.978913    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005"} err="failed to get container status \"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005\": rpc error: code = NotFound desc = could not find container \"bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005\": container with ID starting with bc2f1865cff319b922d0e3816e90633c64daf3dc63399dd411b56edcecd20005 not found: ID does not exist"
Oct 31 00:17:37.034529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034491    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.034722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034588    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.034722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034713    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\" (UniqueName: \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.034839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034714    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.034839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034799    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.034839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034829    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\" (UniqueName: \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.035010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.034874    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\") pod \"54ed9707-5cc1-476c-b4f6-1659f402d75b\" (UID: \"54ed9707-5cc1-476c-b4f6-1659f402d75b\") "
Oct 31 00:17:37.035176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035149    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.035362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035339    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.035648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035624    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.035648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035647    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52aaf67b-7519-484b-af54-a512508d32ed-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.035813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035656    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.035813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035666    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkgws\" (UniqueName: \"kubernetes.io/projected/52aaf67b-7519-484b-af54-a512508d32ed-kube-api-access-wkgws\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.035813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035674    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52aaf67b-7519-484b-af54-a512508d32ed-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.035813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.035682    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.036784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.036752    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.037249 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.037231    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:37.037349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.037330    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz" pod "54ed9707-5cc1-476c-b4f6-1659f402d75b" (UID: "54ed9707-5cc1-476c-b4f6-1659f402d75b"). InnerVolumeSpecName "kube-api-access-dj4zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:37.118546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.118514    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 31 00:17:37.137684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.137643    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54ed9707-5cc1-476c-b4f6-1659f402d75b-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.137684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.137682    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54ed9707-5cc1-476c-b4f6-1659f402d75b-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.137684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.137694    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj4zz\" (UniqueName: \"kubernetes.io/projected/54ed9707-5cc1-476c-b4f6-1659f402d75b-kube-api-access-dj4zz\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.148806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.148770    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:17:37.152912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.152875    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/distinct-2-w2-27e82b-kserve-68bc8f5f94-gnmx4"]
Oct 31 00:17:37.238297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238264    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238344    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238426    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\" (UniqueName: \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238477    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238510    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238608    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.238809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238744    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\") pod \"c5d19655-4df5-42bb-aca8-d782c33fa3df\" (UID: \"c5d19655-4df5-42bb-aca8-d782c33fa3df\") "
Oct 31 00:17:37.238809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238747    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.238954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.238928    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.239401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.239380    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.239543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.239405    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.239543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.239419    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.240488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.240433    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:37.240694 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.240667    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:37.240970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.240945    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf" pod "c5d19655-4df5-42bb-aca8-d782c33fa3df" (UID: "c5d19655-4df5-42bb-aca8-d782c33fa3df"). InnerVolumeSpecName "kube-api-access-mvngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:37.341050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.341002    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d19655-4df5-42bb-aca8-d782c33fa3df-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.341050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.341051    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvngf\" (UniqueName: \"kubernetes.io/projected/c5d19655-4df5-42bb-aca8-d782c33fa3df-kube-api-access-mvngf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.341314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.341069    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d19655-4df5-42bb-aca8-d782c33fa3df-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.423616 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.423569    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 31 00:17:37.452141 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.452095    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/simulator-w2-27e82b-kserve-776b66d97f-p77pd"]
Oct 31 00:17:37.831214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.831174    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52aaf67b-7519-484b-af54-a512508d32ed" path="/var/lib/kubelet/pods/52aaf67b-7519-484b-af54-a512508d32ed/volumes"
Oct 31 00:17:37.831869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.831847    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" path="/var/lib/kubelet/pods/54ed9707-5cc1-476c-b4f6-1659f402d75b/volumes"
Oct 31 00:17:37.955808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.955778    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"
Oct 31 00:17:37.955808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.955789    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn" event={"ID":"c5d19655-4df5-42bb-aca8-d782c33fa3df","Type":"ContainerDied","Data":"7851cc97b66b898c1ef3b0ca8273843c712a29dc9312d05027c947305669ae0f"}
Oct 31 00:17:37.956103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:37.955836    6033 scope.go:122] "RemoveContainer" containerID="1e2921034f5f34810e64757ac698b3fd748019a3999116cb75a508ad8093d500"
Oct 31 00:17:38.112676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:38.112580    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 31 00:17:38.118120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:38.118064    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-27e82b/premium-w2-27e82b-kserve-5654b77f46-977rn"]
Oct 31 00:17:39.830679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:39.830635    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" path="/var/lib/kubelet/pods/c5d19655-4df5-42bb-aca8-d782c33fa3df/volumes"
Oct 31 00:17:44.539321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:44.539282    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"]
Oct 31 00:17:44.585844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:44.585803    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9c21e-5gqjf"]
Oct 31 00:17:45.831258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:45.831219    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71abc5d3-394d-4917-8fd7-094367dd4e51" path="/var/lib/kubelet/pods/71abc5d3-394d-4917-8fd7-094367dd4e51/volumes"
Oct 31 00:17:46.971620 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:17:46.971586    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:17:50.768551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:50.768510    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"]
Oct 31 00:17:50.814994 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:50.814952    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4a3dba-rmj4k"]
Oct 31 00:17:51.846044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:17:51.845985    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa6042e-174b-46ac-af63-a59dc462776f" path="/var/lib/kubelet/pods/9fa6042e-174b-46ac-af63-a59dc462776f/volumes"
Oct 31 00:18:10.733848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.733811    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"]
Oct 31 00:18:10.734483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734469    6033 cpu_manager.go:414] "containerMap: removing container" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main"
Oct 31 00:18:10.734529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734483    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main"
Oct 31 00:18:10.734529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734510    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main"
Oct 31 00:18:10.734529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734515    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main"
Oct 31 00:18:10.734529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734524    6033 cpu_manager.go:414] "containerMap: removing container" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main"
Oct 31 00:18:10.734529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734529    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734543    6033 cpu_manager.go:414] "containerMap: removing container" podUID="71abc5d3-394d-4917-8fd7-094367dd4e51" containerName="revoke-keys"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734548    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71abc5d3-394d-4917-8fd7-094367dd4e51" containerName="revoke-keys"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734558    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fbc4c009-958c-43e9-a338-6af211f423e5" containerName="revoke-keys"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734562    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc4c009-958c-43e9-a338-6af211f423e5" containerName="revoke-keys"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734692    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d19655-4df5-42bb-aca8-d782c33fa3df" containerName="main"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734707    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="52aaf67b-7519-484b-af54-a512508d32ed" containerName="main"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734720    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="71abc5d3-394d-4917-8fd7-094367dd4e51" containerName="revoke-keys"
Oct 31 00:18:10.734727 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734729    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc4c009-958c-43e9-a338-6af211f423e5" containerName="revoke-keys"
Oct 31 00:18:10.734970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.734745    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ed9707-5cc1-476c-b4f6-1659f402d75b" containerName="main"
Oct 31 00:18:10.740108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.740080    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.793495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.793432    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"]
Oct 31 00:18:10.842488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.842444    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.842670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.842567    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.842670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.842663    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.850624 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.850578    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:10.896048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.895996    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:18:10.944479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.944433    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.944665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.944523    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.944709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.944676    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.945344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.945324    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:10.947087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:10.947060    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:11.003158 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:11.003126    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\") pod \"maas-api-revoke-sub-257e4c78-wvtxl\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:11.056128 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:11.056069    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:11.440088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:11.440055    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"]
Oct 31 00:18:11.441881 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:18:11.441602    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0588f8_9a1e_44f3_af1c_b5f3aa5895d6.slice/crio-b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29 WatchSource:0}: Error finding container b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29: Status 404 returned error can't find the container with id b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29
Oct 31 00:18:12.280967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:12.280921    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl" event={"ID":"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6","Type":"ContainerStarted","Data":"cb4a21f07a67220287874d7d681f283f293d70054fce62c41a823baedcfd4aa2"}
Oct 31 00:18:12.281403 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:12.281187    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl" event={"ID":"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6","Type":"ContainerStarted","Data":"b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29"}
Oct 31 00:18:12.397971 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:12.397911    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl" podStartSLOduration=2.397894481 podStartE2EDuration="2.397894481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:12.39524385 +0000 UTC m=+2144.742227748" watchObservedRunningTime="2026-10-31 00:18:12.397894481 +0000 UTC m=+2144.744878410"
Oct 31 00:18:13.290942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:13.290907    6033 generic.go:356] "Generic (PLEG): container finished" podID="2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" containerID="cb4a21f07a67220287874d7d681f283f293d70054fce62c41a823baedcfd4aa2" exitCode=0
Oct 31 00:18:13.291398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:13.290986    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl" event={"ID":"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6","Type":"ContainerDied","Data":"cb4a21f07a67220287874d7d681f283f293d70054fce62c41a823baedcfd4aa2"}
Oct 31 00:18:14.461424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.461392    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:14.598276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.598190    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\") pod \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") "
Oct 31 00:18:14.598276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.598264    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\") pod \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") "
Oct 31 00:18:14.598507 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.598489    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\") pod \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\" (UID: \"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6\") "
Oct 31 00:18:14.598819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.598794    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca" pod "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" (UID: "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:14.599108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.599090    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.600713 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.600684    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd" pod "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" (UID: "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6"). InnerVolumeSpecName "kube-api-access-848vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:14.601099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.601072    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca" pod "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" (UID: "2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:14.700492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.700448    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848vd\" (UniqueName: \"kubernetes.io/projected/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-kube-api-access-848vd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:14.700492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:14.700487    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:15.313073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:15.312147    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"
Oct 31 00:18:15.313073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:15.312165    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl" event={"ID":"2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6","Type":"ContainerDied","Data":"b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29"}
Oct 31 00:18:15.313073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:15.312233    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b61cc31e120272a7c8134d5c9c2575ff6677ab93f68139f66de63d2fa5d6da29"
Oct 31 00:18:16.970836 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:18:16.970795    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:18:20.780439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:20.780399    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:18:25.016323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.016288    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:18:25.016729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.016530    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" podUID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" containerName="maas-api" containerID="cri-o://7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d" gracePeriod=30
Oct 31 00:18:25.387800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.387762    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:18:25.391690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.391654    6033 generic.go:356] "Generic (PLEG): container finished" podID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" containerID="7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d" exitCode=0
Oct 31 00:18:25.391895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.391724    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"
Oct 31 00:18:25.391895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.391743    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" event={"ID":"271e1e35-91a4-4c9e-9440-3e2e6d0257a8","Type":"ContainerDied","Data":"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"}
Oct 31 00:18:25.391895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.391799    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc" event={"ID":"271e1e35-91a4-4c9e-9440-3e2e6d0257a8","Type":"ContainerDied","Data":"1909c3096db5b0629087c81e4621f5e5ad8c116f2e807913fea44f4321f2cad9"}
Oct 31 00:18:25.391895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.391824    6033 scope.go:122] "RemoveContainer" containerID="7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"
Oct 31 00:18:25.404074 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.404043    6033 scope.go:122] "RemoveContainer" containerID="7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"
Oct 31 00:18:25.404486 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:18:25.404459    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d\": container with ID starting with 7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d not found: ID does not exist" containerID="7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"
Oct 31 00:18:25.404573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.404501    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d"} err="failed to get container status \"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d\": rpc error: code = NotFound desc = could not find container \"7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d\": container with ID starting with 7c97da2c20b7050ba671540b54a3c0a594ffdbb7dd2dfa06f680a35047b2477d not found: ID does not exist"
Oct 31 00:18:25.524525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.524483    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\" (UniqueName: \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\") pod \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") "
Oct 31 00:18:25.524733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.524587    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\") pod \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") "
Oct 31 00:18:25.524733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.524643    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\") pod \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") "
Oct 31 00:18:25.524824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.524754    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") pod \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\" (UID: \"271e1e35-91a4-4c9e-9440-3e2e6d0257a8\") "
Oct 31 00:18:25.533441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.533121    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca" pod "271e1e35-91a4-4c9e-9440-3e2e6d0257a8" (UID: "271e1e35-91a4-4c9e-9440-3e2e6d0257a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:25.576725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.558387    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls" pod "271e1e35-91a4-4c9e-9440-3e2e6d0257a8" (UID: "271e1e35-91a4-4c9e-9440-3e2e6d0257a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:25.595051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.593300    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs" pod "271e1e35-91a4-4c9e-9440-3e2e6d0257a8" (UID: "271e1e35-91a4-4c9e-9440-3e2e6d0257a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:25.607449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.607353    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq" pod "271e1e35-91a4-4c9e-9440-3e2e6d0257a8" (UID: "271e1e35-91a4-4c9e-9440-3e2e6d0257a8"). InnerVolumeSpecName "kube-api-access-v64jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:25.637398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.635798    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.637398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.635856    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64jq\" (UniqueName: \"kubernetes.io/projected/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-kube-api-access-v64jq\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.637398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.635873    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.637398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.635888    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/271e1e35-91a4-4c9e-9440-3e2e6d0257a8-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:25.808513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.808474    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:18:25.812344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.812312    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a251fdfb-9c4d7db57-9q2vc"]
Oct 31 00:18:25.830408 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:25.830368    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" path="/var/lib/kubelet/pods/271e1e35-91a4-4c9e-9440-3e2e6d0257a8/volumes"
Oct 31 00:18:27.156132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:27.156086    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"]
Oct 31 00:18:27.219054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:27.218198    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a251fdfb-cxrtz"]
Oct 31 00:18:27.830427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:27.830393    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a64369-d4f7-400e-adb0-c6b6af995559" path="/var/lib/kubelet/pods/a4a64369-d4f7-400e-adb0-c6b6af995559/volumes"
Oct 31 00:18:28.445831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:28.445789    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"]
Oct 31 00:18:28.492368 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:28.492325    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03c703-4v79q"]
Oct 31 00:18:28.780587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:28.780553    6033 scope.go:122] "RemoveContainer" containerID="9a782b791a76cd78fa9575872c565cce2a0ee7fe19642dfb9bb122c45c17a4f2"
Oct 31 00:18:28.791296 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:28.791266    6033 scope.go:122] "RemoveContainer" containerID="e225faadfc0059d8ec5febf020f65699bba31111be49059145a00d5bd3f68211"
Oct 31 00:18:29.830441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:29.830397    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae08413-7163-4610-b00a-7746990ec658" path="/var/lib/kubelet/pods/4ae08413-7163-4610-b00a-7746990ec658/volumes"
Oct 31 00:18:36.051464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.051420    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"]
Oct 31 00:18:36.052533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052486    6033 cpu_manager.go:414] "containerMap: removing container" podUID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" containerName="maas-api"
Oct 31 00:18:36.052533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052515    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" containerName="maas-api"
Oct 31 00:18:36.052631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052554    6033 cpu_manager.go:414] "containerMap: removing container" podUID="2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" containerName="revoke-keys"
Oct 31 00:18:36.052631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052563    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" containerName="revoke-keys"
Oct 31 00:18:36.052780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052764    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="271e1e35-91a4-4c9e-9440-3e2e6d0257a8" containerName="maas-api"
Oct 31 00:18:36.052817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.052786    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" containerName="revoke-keys"
Oct 31 00:18:36.058979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.058950    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.107978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.107942    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"]
Oct 31 00:18:36.119734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.116148    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:36.131097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.131017    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.131343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.131317    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7ff\" (UniqueName: \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.131515 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.131494    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.150642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.150613    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:18:36.232909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.232871    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.233160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.232960    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.233240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.233191    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7ff\" (UniqueName: \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.233655 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.233626    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.235567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.235545    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.293714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.293682    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7ff\" (UniqueName: \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\") pod \"maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.373765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.373673    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:36.754235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:36.754203    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"]
Oct 31 00:18:36.755905 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:18:36.755877    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89afdfb9_e639_4414_b62b_8cf28385319f.slice/crio-d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4 WatchSource:0}: Error finding container d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4: Status 404 returned error can't find the container with id d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4
Oct 31 00:18:37.508009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:37.507971    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp" event={"ID":"89afdfb9-e639-4414-b62b-8cf28385319f","Type":"ContainerStarted","Data":"7f510d8d175dc8b1438a8c06264d58e99355a3dabcea61eede1983e3db2eb39b"}
Oct 31 00:18:37.508492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:37.508048    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp" event={"ID":"89afdfb9-e639-4414-b62b-8cf28385319f","Type":"ContainerStarted","Data":"d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4"}
Oct 31 00:18:37.619374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:37.619313    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp" podStartSLOduration=1.619297235 podStartE2EDuration="1.619297235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:37.616805963 +0000 UTC m=+2169.963789862" watchObservedRunningTime="2026-10-31 00:18:37.619297235 +0000 UTC m=+2169.966281190"
Oct 31 00:18:38.516519 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:38.516483    6033 generic.go:356] "Generic (PLEG): container finished" podID="89afdfb9-e639-4414-b62b-8cf28385319f" containerID="7f510d8d175dc8b1438a8c06264d58e99355a3dabcea61eede1983e3db2eb39b" exitCode=0
Oct 31 00:18:38.516935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:38.516573    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp" event={"ID":"89afdfb9-e639-4414-b62b-8cf28385319f","Type":"ContainerDied","Data":"7f510d8d175dc8b1438a8c06264d58e99355a3dabcea61eede1983e3db2eb39b"}
Oct 31 00:18:39.686788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.686755    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:39.790532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.790439    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\") pod \"89afdfb9-e639-4414-b62b-8cf28385319f\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") "
Oct 31 00:18:39.790680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.790623    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\") pod \"89afdfb9-e639-4414-b62b-8cf28385319f\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") "
Oct 31 00:18:39.790779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.790767    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\" (UniqueName: \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\") pod \"89afdfb9-e639-4414-b62b-8cf28385319f\" (UID: \"89afdfb9-e639-4414-b62b-8cf28385319f\") "
Oct 31 00:18:39.790815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.790775    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca" pod "89afdfb9-e639-4414-b62b-8cf28385319f" (UID: "89afdfb9-e639-4414-b62b-8cf28385319f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:39.791151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.791129    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:39.792796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.792764    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca" pod "89afdfb9-e639-4414-b62b-8cf28385319f" (UID: "89afdfb9-e639-4414-b62b-8cf28385319f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:39.792884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.792848    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff" pod "89afdfb9-e639-4414-b62b-8cf28385319f" (UID: "89afdfb9-e639-4414-b62b-8cf28385319f"). InnerVolumeSpecName "kube-api-access-wn7ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:39.892692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.892640    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89afdfb9-e639-4414-b62b-8cf28385319f-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:39.892692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:39.892682    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7ff\" (UniqueName: \"kubernetes.io/projected/89afdfb9-e639-4414-b62b-8cf28385319f-kube-api-access-wn7ff\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:39.911611 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:18:39.911571    6033 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89afdfb9_e639_4414_b62b_8cf28385319f.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:18:40.532196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:40.532158    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp" event={"ID":"89afdfb9-e639-4414-b62b-8cf28385319f","Type":"ContainerDied","Data":"d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4"}
Oct 31 00:18:40.532196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:40.532188    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"
Oct 31 00:18:40.532462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:40.532211    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d54847ae1647bc0bc63b77ed28cb382052691cbc47a76716d0c689b22d4acbc4"
Oct 31 00:18:41.947842 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:41.947799    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:18:41.948893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:41.948834    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" podUID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" containerName="maas-api" containerID="cri-o://4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b" gracePeriod=30
Oct 31 00:18:42.318147 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.318112    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:18:42.441382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.441343    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\" (UniqueName: \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\") pod \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") "
Oct 31 00:18:42.441662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.441636    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") pod \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") "
Oct 31 00:18:42.441766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.441734    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\") pod \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") "
Oct 31 00:18:42.441845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.441786    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\") pod \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\" (UID: \"5b1d2b89-0c87-4201-a564-d3a02da3a5d6\") "
Oct 31 00:18:42.443587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.443557    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca" pod "5b1d2b89-0c87-4201-a564-d3a02da3a5d6" (UID: "5b1d2b89-0c87-4201-a564-d3a02da3a5d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:42.454221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.448498    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs" pod "5b1d2b89-0c87-4201-a564-d3a02da3a5d6" (UID: "5b1d2b89-0c87-4201-a564-d3a02da3a5d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:42.481963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.481679    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls" pod "5b1d2b89-0c87-4201-a564-d3a02da3a5d6" (UID: "5b1d2b89-0c87-4201-a564-d3a02da3a5d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:42.493090 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.482352    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg" pod "5b1d2b89-0c87-4201-a564-d3a02da3a5d6" (UID: "5b1d2b89-0c87-4201-a564-d3a02da3a5d6"). InnerVolumeSpecName "kube-api-access-8kcfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:42.543606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.543509    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:42.543606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.543547    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:42.543606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.543556    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:42.543606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.543565    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcfg\" (UniqueName: \"kubernetes.io/projected/5b1d2b89-0c87-4201-a564-d3a02da3a5d6-kube-api-access-8kcfg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:18:42.550357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.550320    6033 generic.go:356] "Generic (PLEG): container finished" podID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" containerID="4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b" exitCode=0
Oct 31 00:18:42.550540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.550364    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" event={"ID":"5b1d2b89-0c87-4201-a564-d3a02da3a5d6","Type":"ContainerDied","Data":"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"}
Oct 31 00:18:42.550540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.550374    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"
Oct 31 00:18:42.550540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.550417    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx" event={"ID":"5b1d2b89-0c87-4201-a564-d3a02da3a5d6","Type":"ContainerDied","Data":"92d25a2a5abd16ec0f30063c1b70ad93ae09e0b8655c080cb70bd4aa1ebe4812"}
Oct 31 00:18:42.550540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.550442    6033 scope.go:122] "RemoveContainer" containerID="4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"
Oct 31 00:18:42.587323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.587213    6033 scope.go:122] "RemoveContainer" containerID="4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"
Oct 31 00:18:42.587725 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:18:42.587688    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b\": container with ID starting with 4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b not found: ID does not exist" containerID="4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"
Oct 31 00:18:42.587865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.587739    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b"} err="failed to get container status \"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b\": rpc error: code = NotFound desc = could not find container \"4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b\": container with ID starting with 4497f253b03bc845a375ec3a867f022726175e9e602e136e5246a588a858a19b not found: ID does not exist"
Oct 31 00:18:42.707126 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.707088    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:18:42.711465 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:42.711426    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d9e782-6644dff5f9-xx2qx"]
Oct 31 00:18:43.831097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:43.831054    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" path="/var/lib/kubelet/pods/5b1d2b89-0c87-4201-a564-d3a02da3a5d6/volumes"
Oct 31 00:18:44.136195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:44.136104    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"]
Oct 31 00:18:44.183207 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:44.183171    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d9e782-9gb2t"]
Oct 31 00:18:45.830354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:18:45.830309    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc4c009-958c-43e9-a338-6af211f423e5" path="/var/lib/kubelet/pods/fbc4c009-958c-43e9-a338-6af211f423e5/volumes"
Oct 31 00:18:46.973380 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:18:46.973338    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:19:16.972592 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:16.972550    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:19:17.469114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.469076    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:19:17.469713 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.469672    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy" containerID="cri-o://ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626" gracePeriod=30
Oct 31 00:19:17.491429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.491388    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:19:17.491703 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.491666    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy" containerID="cri-o://7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524" gracePeriod=30
Oct 31 00:19:17.510077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.510011    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:19:17.510275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.510252    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy" containerID="cri-o://8c5a5d5529deac486dc6aabca1219fe46beb513921a66c8492696be475c6444a" gracePeriod=30
Oct 31 00:19:17.515233 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.515198    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:19:17.515428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.515405    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery" containerID="cri-o://53ca2a6906d9dc5c0ab979b0232510e7e10cf63fea1fb29aa243728f0a30430d" gracePeriod=30
Oct 31 00:19:17.891709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.891670    6033 generic.go:356] "Generic (PLEG): container finished" podID="355ca3df-4302-4a5f-8712-f4299e910319" containerID="53ca2a6906d9dc5c0ab979b0232510e7e10cf63fea1fb29aa243728f0a30430d" exitCode=0
Oct 31 00:19:17.891925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.891747    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" event={"ID":"355ca3df-4302-4a5f-8712-f4299e910319","Type":"ContainerDied","Data":"53ca2a6906d9dc5c0ab979b0232510e7e10cf63fea1fb29aa243728f0a30430d"}
Oct 31 00:19:17.944357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:17.944325    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:19:18.106214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106174    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106220    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106276    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106375    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106434    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106528    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106614    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\") pod \"355ca3df-4302-4a5f-8712-f4299e910319\" (UID: \"355ca3df-4302-4a5f-8712-f4299e910319\") "
Oct 31 00:19:18.106790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.106721    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:18.107347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.107295    6033 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-ca-configmap\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.108777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.108740    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:18.108897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.108849    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:18.109149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.109129    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:18.109338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.109310    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:18.109338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.109331    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "kube-api-access-npw8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:18.109517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.109316    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token" pod "355ca3df-4302-4a5f-8712-f4299e910319" (UID: "355ca3df-4302-4a5f-8712-f4299e910319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:18.208712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208668    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.208712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208707    6033 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-csr-dns-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.208712 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208717    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npw8p\" (UniqueName: \"kubernetes.io/projected/355ca3df-4302-4a5f-8712-f4299e910319-kube-api-access-npw8p\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.208975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208727    6033 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-istio-kubeconfig\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.208975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208737    6033 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/355ca3df-4302-4a5f-8712-f4299e910319-cacerts\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.208975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.208746    6033 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/355ca3df-4302-4a5f-8712-f4299e910319-local-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:18.903170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.903139    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"
Oct 31 00:19:18.903373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.903166    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph" event={"ID":"355ca3df-4302-4a5f-8712-f4299e910319","Type":"ContainerDied","Data":"fae8bfd6a7046345b1eac06849b965304d44816e850ac8c25cd7f18a6b2121ba"}
Oct 31 00:19:18.903373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:18.903232    6033 scope.go:122] "RemoveContainer" containerID="53ca2a6906d9dc5c0ab979b0232510e7e10cf63fea1fb29aa243728f0a30430d"
Oct 31 00:19:19.085286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:19.085242    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:19:19.133078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:19.133041    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzzph"]
Oct 31 00:19:19.834086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:19.834051    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355ca3df-4302-4a5f-8712-f4299e910319" path="/var/lib/kubelet/pods/355ca3df-4302-4a5f-8712-f4299e910319/volumes"
Oct 31 00:19:22.473972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.473933    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 31 00:19:22.474459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.474171    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy" containerID="cri-o://30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe" gracePeriod=30
Oct 31 00:19:22.879214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.879183    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:19:22.935309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.935277    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:19:22.936551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.936523    6033 generic.go:356] "Generic (PLEG): container finished" podID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerID="8c5a5d5529deac486dc6aabca1219fe46beb513921a66c8492696be475c6444a" exitCode=0
Oct 31 00:19:22.936674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.936598    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" event={"ID":"ac19bf16-b851-4d0c-bb65-ef442299f0b2","Type":"ContainerDied","Data":"8c5a5d5529deac486dc6aabca1219fe46beb513921a66c8492696be475c6444a"}
Oct 31 00:19:22.938362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.938333    6033 generic.go:356] "Generic (PLEG): container finished" podID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerID="ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626" exitCode=0
Oct 31 00:19:22.938509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.938393    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"
Oct 31 00:19:22.938589 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.938522    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" event={"ID":"7338b82b-a26e-4f3d-8ccf-c283e7218542","Type":"ContainerDied","Data":"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"}
Oct 31 00:19:22.938589 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.938559    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w" event={"ID":"7338b82b-a26e-4f3d-8ccf-c283e7218542","Type":"ContainerDied","Data":"56002733c0878139be540e628efbb33c04b0a5d6406bb82458717b0ed9fc4edc"}
Oct 31 00:19:22.938589 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.938585    6033 scope.go:122] "RemoveContainer" containerID="ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"
Oct 31 00:19:22.940452 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.940358    6033 generic.go:356] "Generic (PLEG): container finished" podID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerID="7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524" exitCode=0
Oct 31 00:19:22.940524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.940459    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"
Oct 31 00:19:22.940710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.940692    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" event={"ID":"f0fa903e-7b55-45c8-abde-0f613c8a7d77","Type":"ContainerDied","Data":"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"}
Oct 31 00:19:22.940766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.940737    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc" event={"ID":"f0fa903e-7b55-45c8-abde-0f613c8a7d77","Type":"ContainerDied","Data":"0237d9e0aa2cb91310bcc803470be450d5f195031046ca819c04c55878aba01d"}
Oct 31 00:19:22.951180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.951153    6033 scope.go:122] "RemoveContainer" containerID="ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"
Oct 31 00:19:22.951602 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:22.951534    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626\": container with ID starting with ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626 not found: ID does not exist" containerID="ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"
Oct 31 00:19:22.951602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.951566    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626"} err="failed to get container status \"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626\": rpc error: code = NotFound desc = could not find container \"ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626\": container with ID starting with ce388fc4c47bcb228d101291ce0615f96f3a403c557abb378dc48f71cb08a626 not found: ID does not exist"
Oct 31 00:19:22.951602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.951588    6033 scope.go:122] "RemoveContainer" containerID="7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"
Oct 31 00:19:22.962552 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.962513    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:19:22.963808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.963779    6033 scope.go:122] "RemoveContainer" containerID="7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"
Oct 31 00:19:22.964184 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:22.964160    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524\": container with ID starting with 7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524 not found: ID does not exist" containerID="7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"
Oct 31 00:19:22.964263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.964195    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524"} err="failed to get container status \"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524\": rpc error: code = NotFound desc = could not find container \"7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524\": container with ID starting with 7f51c9dbd531e4f52bf50c425ce345a33faaae8b2d9e67f9205bac389be53524 not found: ID does not exist"
Oct 31 00:19:22.980824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.980788    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.980979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.980832    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.980979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.980865    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.980979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.980902    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.980979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.980957    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981080    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.981191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981137    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981174    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981206    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.981354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981239    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.981354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981275    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981305    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.981581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981410    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.981581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981442    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981470    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981525    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.981581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981541    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981595    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\") pod \"7338b82b-a26e-4f3d-8ccf-c283e7218542\" (UID: \"7338b82b-a26e-4f3d-8ccf-c283e7218542\") "
Oct 31 00:19:22.981835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981595    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.981835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981696    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:22.981996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981907    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:22.981996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.981918    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.982201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.982173    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.982327 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.982201    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\") pod \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\" (UID: \"f0fa903e-7b55-45c8-abde-0f613c8a7d77\") "
Oct 31 00:19:22.982385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.982371    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.982544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.982523    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.982963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.982940    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983128    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.983193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983140    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983154    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.983193 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983161    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983216    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983235    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983252    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983266    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.983480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.983281    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338b82b-a26e-4f3d-8ccf-c283e7218542-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:22.984739 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.984710    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.985497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.985431    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "kube-api-access-8nmxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:22.985711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.985681    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:22.985951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.985909    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:22.986157 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.986135    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo" pod "f0fa903e-7b55-45c8-abde-0f613c8a7d77" (UID: "f0fa903e-7b55-45c8-abde-0f613c8a7d77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:22.986437 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.986404    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:22.986956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.986932    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:22.987183 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:22.987163    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb" pod "7338b82b-a26e-4f3d-8ccf-c283e7218542" (UID: "7338b82b-a26e-4f3d-8ccf-c283e7218542"). InnerVolumeSpecName "kube-api-access-527kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.084995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.084694    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.084995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.084819    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.085978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.084872    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086229    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086294    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086338    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086476    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086627    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.086670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.086660    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\") pod \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\" (UID: \"ac19bf16-b851-4d0c-bb65-ef442299f0b2\") "
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087273    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087297    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmxk\" (UniqueName: \"kubernetes.io/projected/f0fa903e-7b55-45c8-abde-0f613c8a7d77-kube-api-access-8nmxk\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087308    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527kb\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-kube-api-access-527kb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087319    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087328    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087336    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087346    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa903e-7b55-45c8-abde-0f613c8a7d77-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087356    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087369    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.087793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.087379    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338b82b-a26e-4f3d-8ccf-c283e7218542-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.088791 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.088762    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.088891 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.088781    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.088954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.088879    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.089016 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.088994    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.089396 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.089371    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:23.092207 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.092175    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:23.130659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.130617    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:23.130930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.130802    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "kube-api-access-h8vh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.133102 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.133049    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token" pod "ac19bf16-b851-4d0c-bb65-ef442299f0b2" (UID: "ac19bf16-b851-4d0c-bb65-ef442299f0b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:23.189226 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189183    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189226 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189229    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189244    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189259    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189273    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189290    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189303    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189315    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19bf16-b851-4d0c-bb65-ef442299f0b2-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.189458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.189329    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vh9\" (UniqueName: \"kubernetes.io/projected/ac19bf16-b851-4d0c-bb65-ef442299f0b2-kube-api-access-h8vh9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:23.383173 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.383040    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:19:23.387005 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.386953    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-j9skc"]
Oct 31 00:19:23.555339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.535656    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:19:23.555339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.540534    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-q6d2w"]
Oct 31 00:19:23.840455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.840404    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" path="/var/lib/kubelet/pods/7338b82b-a26e-4f3d-8ccf-c283e7218542/volumes"
Oct 31 00:19:23.841434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.841405    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" path="/var/lib/kubelet/pods/f0fa903e-7b55-45c8-abde-0f613c8a7d77/volumes"
Oct 31 00:19:23.950536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.949723    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"
Oct 31 00:19:23.950536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.949742    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k" event={"ID":"ac19bf16-b851-4d0c-bb65-ef442299f0b2","Type":"ContainerDied","Data":"cf11528ce10a42d396b1890f71386da6f03eac374b92b7d72cf79b32291a5ab5"}
Oct 31 00:19:23.950536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:23.949827    6033 scope.go:122] "RemoveContainer" containerID="8c5a5d5529deac486dc6aabca1219fe46beb513921a66c8492696be475c6444a"
Oct 31 00:19:24.110954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:24.110709    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:19:24.115529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:24.115488    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a251fdfb-openshift-default-7f7f49cc57-z9l6k"]
Oct 31 00:19:24.562086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:24.562050    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"]
Oct 31 00:19:24.567038 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:24.566995    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-27e82b-s8sxw"]
Oct 31 00:19:25.736243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:25.736206    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"]
Oct 31 00:19:25.781776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:25.781739    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-39e3a8-kv4ct"]
Oct 31 00:19:25.831015 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:25.830966    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd836ce-5cda-403c-9ad5-174019f6d5b2" path="/var/lib/kubelet/pods/2bd836ce-5cda-403c-9ad5-174019f6d5b2/volumes"
Oct 31 00:19:25.831600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:25.831576    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ebd360d-f75f-45be-9019-8e1cce769a77" path="/var/lib/kubelet/pods/5ebd360d-f75f-45be-9019-8e1cce769a77/volumes"
Oct 31 00:19:25.832135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:25.832114    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" path="/var/lib/kubelet/pods/ac19bf16-b851-4d0c-bb65-ef442299f0b2/volumes"
Oct 31 00:19:27.744080 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.743991    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 31 00:19:27.878495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878455    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878495 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878503    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878605    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878642    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878672    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878702    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878799    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878852    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878865    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.878978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878944    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\") pod \"04a2b4d1-a073-445a-910c-fb23a8de8faa\" (UID: \"04a2b4d1-a073-445a-910c-fb23a8de8faa\") "
Oct 31 00:19:27.878978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878967    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.879332 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.878984    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.879332 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879152    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.879451 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879329    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:27.879722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879695    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.879722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879724    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.879913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879737    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a2b4d1-a073-445a-910c-fb23a8de8faa-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.879913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879748    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.879913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.879758    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.881247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.881220    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:27.881354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.881323    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:27.881469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.881436    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:27.881634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.881610    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2" pod "04a2b4d1-a073-445a-910c-fb23a8de8faa" (UID: "04a2b4d1-a073-445a-910c-fb23a8de8faa"). InnerVolumeSpecName "kube-api-access-7hlp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:27.981342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.981241    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.981342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.981281    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.981342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.981298    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlp2\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-kube-api-access-7hlp2\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.981342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.981312    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a2b4d1-a073-445a-910c-fb23a8de8faa-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:27.988096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.988045    6033 generic.go:356] "Generic (PLEG): container finished" podID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerID="30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe" exitCode=0
Oct 31 00:19:27.988275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.988124    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"
Oct 31 00:19:27.988275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.988126    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" event={"ID":"04a2b4d1-a073-445a-910c-fb23a8de8faa","Type":"ContainerDied","Data":"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"}
Oct 31 00:19:27.988275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.988197    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp" event={"ID":"04a2b4d1-a073-445a-910c-fb23a8de8faa","Type":"ContainerDied","Data":"52cf5d61eeb32ff21dae83de17c0e0643be020a48fa02f926c2c0424e3f75481"}
Oct 31 00:19:27.988275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:27.988229    6033 scope.go:122] "RemoveContainer" containerID="30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"
Oct 31 00:19:28.003137 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:28.003109    6033 scope.go:122] "RemoveContainer" containerID="30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"
Oct 31 00:19:28.003501 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:28.003482    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe\": container with ID starting with 30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe not found: ID does not exist" containerID="30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"
Oct 31 00:19:28.003557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:28.003513    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe"} err="failed to get container status \"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe\": rpc error: code = NotFound desc = could not find container \"30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe\": container with ID starting with 30bf036257139c35319f7af96fa9c85b227ca01a99194d8ebab685063d44aebe not found: ID does not exist"
Oct 31 00:19:28.146817 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:28.146783    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 31 00:19:28.151644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:28.151605    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-rb7qp"]
Oct 31 00:19:29.831599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:29.831558    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" path="/var/lib/kubelet/pods/04a2b4d1-a073-445a-910c-fb23a8de8faa/volumes"
Oct 31 00:19:31.117010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:31.116971    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:19:31.555012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:31.554972    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 31 00:19:31.555252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:31.555229    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy" containerID="cri-o://d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32" gracePeriod=30
Oct 31 00:19:32.637933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:32.637891    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:19:32.638439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:32.638194    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy" containerID="cri-o://4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306" gracePeriod=30
Oct 31 00:19:36.856832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.856792    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 31 00:19:36.908802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.908765    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.908837    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.908890    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.908958    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.908994    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909105    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909196    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909325    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909374    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\") pod \"418c94f0-9f39-4846-b629-808c772500fc\" (UID: \"418c94f0-9f39-4846-b629-808c772500fc\") "
Oct 31 00:19:36.909498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909393    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:36.909498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909397    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.909614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909531    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.909614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909599    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.909738 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.909714    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.910347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.910323    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.910532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.910512    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.910619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.910535    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.910619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.910562    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/418c94f0-9f39-4846-b629-808c772500fc-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.910619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.910578    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:36.911912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.911888    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:36.911986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.911887    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:36.912181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.912160    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:36.912269 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:36.912248    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74" pod "418c94f0-9f39-4846-b629-808c772500fc" (UID: "418c94f0-9f39-4846-b629-808c772500fc"). InnerVolumeSpecName "kube-api-access-29z74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:37.012385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.012343    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.012385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.012378    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/418c94f0-9f39-4846-b629-808c772500fc-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.012385 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.012389    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/418c94f0-9f39-4846-b629-808c772500fc-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.012695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.012404    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29z74\" (UniqueName: \"kubernetes.io/projected/418c94f0-9f39-4846-b629-808c772500fc-kube-api-access-29z74\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:37.085963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.085860    6033 generic.go:356] "Generic (PLEG): container finished" podID="418c94f0-9f39-4846-b629-808c772500fc" containerID="d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32" exitCode=0
Oct 31 00:19:37.085963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.085953    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"
Oct 31 00:19:37.086254 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.085955    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" event={"ID":"418c94f0-9f39-4846-b629-808c772500fc","Type":"ContainerDied","Data":"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"}
Oct 31 00:19:37.086254 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.086114    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf" event={"ID":"418c94f0-9f39-4846-b629-808c772500fc","Type":"ContainerDied","Data":"3d91aeaecf0ebefe2981906f746401f6dc99e03b0af6a86a83596bb910258f91"}
Oct 31 00:19:37.086254 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.086150    6033 scope.go:122] "RemoveContainer" containerID="d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"
Oct 31 00:19:37.100477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.100450    6033 scope.go:122] "RemoveContainer" containerID="d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"
Oct 31 00:19:37.100867 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:37.100840    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32\": container with ID starting with d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32 not found: ID does not exist" containerID="d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"
Oct 31 00:19:37.100978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.100876    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32"} err="failed to get container status \"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32\": rpc error: code = NotFound desc = could not find container \"d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32\": container with ID starting with d611f31cbeaadc8a2a3746b7552740d856ed57bd6246f3dffe62cf4ef863ad32 not found: ID does not exist"
Oct 31 00:19:37.246522 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.246482    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 31 00:19:37.252704 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.252668    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-27e82b-openshift-default-c6c7c4b77-n2wcf"]
Oct 31 00:19:37.831449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.831408    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418c94f0-9f39-4846-b629-808c772500fc" path="/var/lib/kubelet/pods/418c94f0-9f39-4846-b629-808c772500fc/volumes"
Oct 31 00:19:37.951418 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:37.951385    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:19:38.027885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.027835    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.027896    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.027984    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028089    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028156    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:38.028261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028165    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028228    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028373    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028392    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:38.028674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028641    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028566    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:38.028879 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028714    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\") pod \"e69081c7-fca0-4523-b36b-2d412f6e2807\" (UID: \"e69081c7-fca0-4523-b36b-2d412f6e2807\") "
Oct 31 00:19:38.028879 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.028844    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:38.029266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029240    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:38.029386 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029365    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.029467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029391    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.029467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029408    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69081c7-fca0-4523-b36b-2d412f6e2807-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.029467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029422    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.029467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.029435    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.030547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.030521    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:38.030618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.030532    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:38.030753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.030738    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "kube-api-access-fx57s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:38.030795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.030750    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token" pod "e69081c7-fca0-4523-b36b-2d412f6e2807" (UID: "e69081c7-fca0-4523-b36b-2d412f6e2807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:38.098478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.098426    6033 generic.go:356] "Generic (PLEG): container finished" podID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerID="4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306" exitCode=0
Oct 31 00:19:38.098478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.098472    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"
Oct 31 00:19:38.098748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.098508    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" event={"ID":"e69081c7-fca0-4523-b36b-2d412f6e2807","Type":"ContainerDied","Data":"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"}
Oct 31 00:19:38.098748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.098568    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl" event={"ID":"e69081c7-fca0-4523-b36b-2d412f6e2807","Type":"ContainerDied","Data":"eb8ee5ff55144e7c30532c5b045d621ae2ee6307f5d4bed512769c2c2b37dc8f"}
Oct 31 00:19:38.098748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.098594    6033 scope.go:122] "RemoveContainer" containerID="4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"
Oct 31 00:19:38.113699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.113668    6033 scope.go:122] "RemoveContainer" containerID="4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"
Oct 31 00:19:38.114575 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:38.114539    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306\": container with ID starting with 4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306 not found: ID does not exist" containerID="4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"
Oct 31 00:19:38.114708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.114585    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306"} err="failed to get container status \"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306\": rpc error: code = NotFound desc = could not find container \"4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306\": container with ID starting with 4873a51b624b250ba8e2b059c857699a60a61403961bbdcc44e348fff8a45306 not found: ID does not exist"
Oct 31 00:19:38.131023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.130982    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx57s\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-kube-api-access-fx57s\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.131023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.131021    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.131225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.131048    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.131225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.131063    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69081c7-fca0-4523-b36b-2d412f6e2807-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:38.255311 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.255267    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:19:38.259986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.259950    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-r8xrl"]
Oct 31 00:19:38.590180 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.590088    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:19:38.590338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.590307    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy" containerID="cri-o://6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed" gracePeriod=30
Oct 31 00:19:38.844265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.844169    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:19:38.844437 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.844396    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" containerID="cri-o://c2853076354df537ee8817925a5589100e2958cb28a440bc3acd0a12a95f77a1" gracePeriod=30
Oct 31 00:19:38.848318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.848281    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:19:38.848753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:38.848721    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" containerID="cri-o://f3bd69855c0170b6cb7083b9ce90fb2dbcdd0c56a25152eb3923b057f5907273" gracePeriod=30
Oct 31 00:19:39.837153 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:39.837108    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" path="/var/lib/kubelet/pods/e69081c7-fca0-4523-b36b-2d412f6e2807/volumes"
Oct 31 00:19:41.599533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.599490    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-thj8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:19:41.599991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.599558    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:19:41.655347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.655307    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-wv5pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:19:41.655565 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.655371    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:19:41.799008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.798968    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"]
Oct 31 00:19:41.845111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:41.845076    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-460d4e-x6mjp"]
Oct 31 00:19:43.832783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:43.832743    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89afdfb9-e639-4414-b62b-8cf28385319f" path="/var/lib/kubelet/pods/89afdfb9-e639-4414-b62b-8cf28385319f/volumes"
Oct 31 00:19:44.013734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.013701    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:19:44.107477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107436    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.107666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107653    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107758    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107808    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107836    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107863    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107913    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.107966    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.108004    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\") pod \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\" (UID: \"344316a1-caba-4f26-8d3a-3bcd2a4b89ee\") "
Oct 31 00:19:44.108844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.108814    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.110574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.109023    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.110574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.109910    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.110574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.110233    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:44.110890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.110866    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.112644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.112609    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.128318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.128270    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "kube-api-access-dpjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.129164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.129135    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:44.129287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.129255    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy" pod "344316a1-caba-4f26-8d3a-3bcd2a4b89ee" (UID: "344316a1-caba-4f26-8d3a-3bcd2a4b89ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.145019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.144989    6033 generic.go:356] "Generic (PLEG): container finished" podID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerID="c2853076354df537ee8817925a5589100e2958cb28a440bc3acd0a12a95f77a1" exitCode=0
Oct 31 00:19:44.145183 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.145095    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" event={"ID":"1b3d636a-4076-4198-94ac-ef9bb0db2be1","Type":"ContainerDied","Data":"c2853076354df537ee8817925a5589100e2958cb28a440bc3acd0a12a95f77a1"}
Oct 31 00:19:44.148117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.147170    6033 generic.go:356] "Generic (PLEG): container finished" podID="02390209-e090-495c-841a-78944e9e8f69" containerID="f3bd69855c0170b6cb7083b9ce90fb2dbcdd0c56a25152eb3923b057f5907273" exitCode=0
Oct 31 00:19:44.148117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.147242    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" event={"ID":"02390209-e090-495c-841a-78944e9e8f69","Type":"ContainerDied","Data":"f3bd69855c0170b6cb7083b9ce90fb2dbcdd0c56a25152eb3923b057f5907273"}
Oct 31 00:19:44.152119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.149880    6033 generic.go:356] "Generic (PLEG): container finished" podID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerID="6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed" exitCode=0
Oct 31 00:19:44.152119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.149943    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" event={"ID":"344316a1-caba-4f26-8d3a-3bcd2a4b89ee","Type":"ContainerDied","Data":"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"}
Oct 31 00:19:44.152119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.149992    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67" event={"ID":"344316a1-caba-4f26-8d3a-3bcd2a4b89ee","Type":"ContainerDied","Data":"8ed7e62b4b8fc6a056e60b45663e83bde6803c9c319d43e246cd87b6b9a3d8a2"}
Oct 31 00:19:44.152119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.150023    6033 scope.go:122] "RemoveContainer" containerID="6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"
Oct 31 00:19:44.152119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.150214    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"
Oct 31 00:19:44.167895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.167873    6033 scope.go:122] "RemoveContainer" containerID="6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"
Oct 31 00:19:44.168472 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:44.168364    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed\": container with ID starting with 6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed not found: ID does not exist" containerID="6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"
Oct 31 00:19:44.168472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.168407    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed"} err="failed to get container status \"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed\": rpc error: code = NotFound desc = could not find container \"6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed\": container with ID starting with 6e53432c0360c5fdf3fa8d2d2934e33027fe8769c5c29b2bacfce9e6ef910eed not found: ID does not exist"
Oct 31 00:19:44.174529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.174491    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.183659    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184622    6033 cpu_manager.go:414] "containerMap: removing container" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184680    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184719    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184727    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184749    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184758    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184776    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" containerName="maas-api"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184783    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" containerName="maas-api"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184800    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184808    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184823    6033 cpu_manager.go:414] "containerMap: removing container" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184830    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184844    6033 cpu_manager.go:414] "containerMap: removing container" podUID="89afdfb9-e639-4414-b62b-8cf28385319f" containerName="revoke-keys"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184851    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89afdfb9-e639-4414-b62b-8cf28385319f" containerName="revoke-keys"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184866    6033 cpu_manager.go:414] "containerMap: removing container" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184873    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184889    6033 cpu_manager.go:414] "containerMap: removing container" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184899    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184910    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184918    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184936    6033 cpu_manager.go:414] "containerMap: removing container" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.184943    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185118    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69081c7-fca0-4523-b36b-2d412f6e2807" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185135    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1d2b89-0c87-4201-a564-d3a02da3a5d6" containerName="maas-api"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185147    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7338b82b-a26e-4f3d-8ccf-c283e7218542" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185159    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a2b4d1-a073-445a-910c-fb23a8de8faa" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185170    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="89afdfb9-e639-4414-b62b-8cf28385319f" containerName="revoke-keys"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185179    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="418c94f0-9f39-4846-b629-808c772500fc" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185190    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185202    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fa903e-7b55-45c8-abde-0f613c8a7d77" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185213    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="355ca3df-4302-4a5f-8712-f4299e910319" containerName="discovery"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185225    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.185236    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac19bf16-b851-4d0c-bb65-ef442299f0b2" containerName="istio-proxy"
Oct 31 00:19:44.193108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.190918    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.209957 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.209926    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.209994    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210083    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210117    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210185    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210221    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210289    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210337    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.210766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210370    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\") pod \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\" (UID: \"1b3d636a-4076-4198-94ac-ef9bb0db2be1\") "
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210821    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210866    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsvm\" (UniqueName: \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.210976    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211070    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211089    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211106    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211122    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjv9\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-kube-api-access-dpjv9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211137    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211153    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211166    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211180    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211194    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344316a1-caba-4f26-8d3a-3bcd2a4b89ee-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211203    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211634    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.211821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.211688    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.212857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.212006    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:44.214384 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.214359    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.218510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.218480    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "kube-api-access-ddlhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.220168 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.220146    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:44.220522 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.220498    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy" pod "1b3d636a-4076-4198-94ac-ef9bb0db2be1" (UID: "1b3d636a-4076-4198-94ac-ef9bb0db2be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.267590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.261187    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ddf61a\""
Oct 31 00:19:44.267590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.261466    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ddf61a-dockercfg-8qrl7\""
Oct 31 00:19:44.267590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.264989    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:19:44.288050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.287204    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:19:44.302043 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.301986    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:19:44.305653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.305611    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-qxg67"]
Oct 31 00:19:44.310793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.310741    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:19:44.311771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.311741    6033 cpu_manager.go:414] "containerMap: removing container" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy"
Oct 31 00:19:44.311899 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.311775    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy"
Oct 31 00:19:44.312012 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.311996    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="02390209-e090-495c-841a-78944e9e8f69" containerName="istio-proxy"
Oct 31 00:19:44.312583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312552    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.312687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312619    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.312687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312668    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.312812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312708    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.312909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312890    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.312972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.312958    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.313428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.313403    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.313630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.313609    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.314539 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.314515    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:44.317471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.316826    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.317687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.317653    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.317687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.317678    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "kube-api-access-s8sqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.318101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.318084    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:19:44.323756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.323728    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.323888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.323773    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.323888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.323792    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\") pod \"02390209-e090-495c-841a-78944e9e8f69\" (UID: \"02390209-e090-495c-841a-78944e9e8f69\") "
Oct 31 00:19:44.324063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324022    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.324150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324086    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsvm\" (UniqueName: \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.324475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324453    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8sqg\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-kube-api-access-s8sqg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324480    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324494    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324508    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324521    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324535    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324550    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02390209-e090-495c-841a-78944e9e8f69-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324564    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324579    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlhr\" (UniqueName: \"kubernetes.io/projected/1b3d636a-4076-4198-94ac-ef9bb0db2be1-kube-api-access-ddlhr\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324593    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324608    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324626    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02390209-e090-495c-841a-78944e9e8f69-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324642    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3d636a-4076-4198-94ac-ef9bb0db2be1-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324655    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3d636a-4076-4198-94ac-ef9bb0db2be1-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.324774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.324669    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.325484 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.325456    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.325695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.325675    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:19:44.326677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.326644    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.327831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.327806    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token" pod "02390209-e090-495c-841a-78944e9e8f69" (UID: "02390209-e090-495c-841a-78944e9e8f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:44.370885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.370833    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:19:44.388605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.388527    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsvm\" (UniqueName: \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\") pod \"payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.425590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.425534    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfbh\" (UniqueName: \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.425892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.425853    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.426048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.425995    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.426048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.426014    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02390209-e090-495c-841a-78944e9e8f69-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.426048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.426044    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02390209-e090-495c-841a-78944e9e8f69-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:44.511689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.511636    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:44.527472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.527431    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfbh\" (UniqueName: \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.527656 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.527640    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.528386 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.528360    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.587796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.586814    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfbh\" (UniqueName: \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\") pod \"payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.634876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.634836    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:44.899593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:44.899561    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:19:44.901585 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:19:44.901543    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c29a153_f0a0_406e_8ce3_ba51e3d3a587.slice/crio-acb0aad59195b36a58dddad483d19dfe0a28e367ffcef3a2f438ed7e89e8606e WatchSource:0}: Error finding container acb0aad59195b36a58dddad483d19dfe0a28e367ffcef3a2f438ed7e89e8606e: Status 404 returned error can't find the container with id acb0aad59195b36a58dddad483d19dfe0a28e367ffcef3a2f438ed7e89e8606e
Oct 31 00:19:45.030389 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.030359    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:19:45.032457 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:19:45.032428    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5029d054_88b7_4c49_8954_0783911fbb4c.slice/crio-901f5ff27614cb4966cc0c6f6a84d1eecf62072b24475a5aada44691fbbc7554 WatchSource:0}: Error finding container 901f5ff27614cb4966cc0c6f6a84d1eecf62072b24475a5aada44691fbbc7554: Status 404 returned error can't find the container with id 901f5ff27614cb4966cc0c6f6a84d1eecf62072b24475a5aada44691fbbc7554
Oct 31 00:19:45.158505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.158413    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf" event={"ID":"1b3d636a-4076-4198-94ac-ef9bb0db2be1","Type":"ContainerDied","Data":"07b857750e477a0021b967800fd17fc9df949dad524f88812defc04418c8f857"}
Oct 31 00:19:45.158505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.158482    6033 scope.go:122] "RemoveContainer" containerID="c2853076354df537ee8817925a5589100e2958cb28a440bc3acd0a12a95f77a1"
Oct 31 00:19:45.158505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.158428    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"
Oct 31 00:19:45.160334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.160300    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" event={"ID":"4c29a153-f0a0-406e-8ce3-ba51e3d3a587","Type":"ContainerStarted","Data":"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"}
Oct 31 00:19:45.160483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.160356    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" event={"ID":"4c29a153-f0a0-406e-8ce3-ba51e3d3a587","Type":"ContainerStarted","Data":"acb0aad59195b36a58dddad483d19dfe0a28e367ffcef3a2f438ed7e89e8606e"}
Oct 31 00:19:45.160483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.160421    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:45.162199 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.162175    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" event={"ID":"5029d054-88b7-4c49-8954-0783911fbb4c","Type":"ContainerStarted","Data":"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"}
Oct 31 00:19:45.162320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.162222    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:19:45.162320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.162246    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" event={"ID":"5029d054-88b7-4c49-8954-0783911fbb4c","Type":"ContainerStarted","Data":"901f5ff27614cb4966cc0c6f6a84d1eecf62072b24475a5aada44691fbbc7554"}
Oct 31 00:19:45.163804 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.163765    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z" event={"ID":"02390209-e090-495c-841a-78944e9e8f69","Type":"ContainerDied","Data":"ab8ac3d6e158ca374042ee26642cdd9d55c3b0ab0858c5fe635316ef6a87bd01"}
Oct 31 00:19:45.163920 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.163777    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"
Oct 31 00:19:45.170257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.170238    6033 scope.go:122] "RemoveContainer" containerID="f3bd69855c0170b6cb7083b9ce90fb2dbcdd0c56a25152eb3923b057f5907273"
Oct 31 00:19:45.316742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.316708    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:19:45.321092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.321063    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-thj8z"]
Oct 31 00:19:45.424406 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.424321    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:19:45.431699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.431669    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-wv5pf"]
Oct 31 00:19:45.539745 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.539688    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" podStartSLOduration=1.539673005 podStartE2EDuration="1.539673005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:45.538364662 +0000 UTC m=+2237.885348560" watchObservedRunningTime="2026-10-31 00:19:45.539673005 +0000 UTC m=+2237.886656902"
Oct 31 00:19:45.650349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.650294    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" podStartSLOduration=1.65027594 podStartE2EDuration="1.65027594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:45.647498449 +0000 UTC m=+2237.994482349" watchObservedRunningTime="2026-10-31 00:19:45.65027594 +0000 UTC m=+2237.997259839"
Oct 31 00:19:45.834312 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.834268    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02390209-e090-495c-841a-78944e9e8f69" path="/var/lib/kubelet/pods/02390209-e090-495c-841a-78944e9e8f69/volumes"
Oct 31 00:19:45.834981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.834957    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3d636a-4076-4198-94ac-ef9bb0db2be1" path="/var/lib/kubelet/pods/1b3d636a-4076-4198-94ac-ef9bb0db2be1/volumes"
Oct 31 00:19:45.835610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:45.835587    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344316a1-caba-4f26-8d3a-3bcd2a4b89ee" path="/var/lib/kubelet/pods/344316a1-caba-4f26-8d3a-3bcd2a4b89ee/volumes"
Oct 31 00:19:46.971294 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:19:46.971254    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:19:50.065734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.065673    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"]
Oct 31 00:19:50.069972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.069947    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.122835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.122803    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:19:50.169673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.169639    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:19:50.189737 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.189693    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"]
Oct 31 00:19:50.206867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.206835    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.207067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.206888    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56msl\" (UniqueName: \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.207118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.207071    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.307952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.307914    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56msl\" (UniqueName: \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.308145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.308066    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.308210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.308195    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.308941 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.308915    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.310792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.310767    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.366955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.366867    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56msl\" (UniqueName: \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.384711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.384672    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:50.767514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:50.767480    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"]
Oct 31 00:19:50.771223 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:19:50.771178    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70ebf65_39f9_496b_be1b_88338ca5e330.slice/crio-cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31 WatchSource:0}: Error finding container cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31: Status 404 returned error can't find the container with id cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31
Oct 31 00:19:51.211983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:51.211942    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc" event={"ID":"c70ebf65-39f9-496b-be1b-88338ca5e330","Type":"ContainerStarted","Data":"4885432e1131277af6841d99b4d58ce43d38f716df2f46fa9f619519a37c44aa"}
Oct 31 00:19:51.212393 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:51.212001    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc" event={"ID":"c70ebf65-39f9-496b-be1b-88338ca5e330","Type":"ContainerStarted","Data":"cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31"}
Oct 31 00:19:51.322894 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:51.322831    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc" podStartSLOduration=1.322816551 podStartE2EDuration="1.322816551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:51.321937007 +0000 UTC m=+2243.668920904" watchObservedRunningTime="2026-10-31 00:19:51.322816551 +0000 UTC m=+2243.669800448"
Oct 31 00:19:52.220991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:52.220900    6033 generic.go:356] "Generic (PLEG): container finished" podID="c70ebf65-39f9-496b-be1b-88338ca5e330" containerID="4885432e1131277af6841d99b4d58ce43d38f716df2f46fa9f619519a37c44aa" exitCode=0
Oct 31 00:19:52.220991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:52.220967    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc" event={"ID":"c70ebf65-39f9-496b-be1b-88338ca5e330","Type":"ContainerDied","Data":"4885432e1131277af6841d99b4d58ce43d38f716df2f46fa9f619519a37c44aa"}
Oct 31 00:19:53.385153 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.385114    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:53.545897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.545856    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\" (UniqueName: \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\") pod \"c70ebf65-39f9-496b-be1b-88338ca5e330\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") "
Oct 31 00:19:53.546099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.545953    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\") pod \"c70ebf65-39f9-496b-be1b-88338ca5e330\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") "
Oct 31 00:19:53.546099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.545986    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\") pod \"c70ebf65-39f9-496b-be1b-88338ca5e330\" (UID: \"c70ebf65-39f9-496b-be1b-88338ca5e330\") "
Oct 31 00:19:53.546416 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.546390    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca" pod "c70ebf65-39f9-496b-be1b-88338ca5e330" (UID: "c70ebf65-39f9-496b-be1b-88338ca5e330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:19:53.546633 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.546614    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:53.548213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.548176    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca" pod "c70ebf65-39f9-496b-be1b-88338ca5e330" (UID: "c70ebf65-39f9-496b-be1b-88338ca5e330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:19:53.548323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.548293    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl" pod "c70ebf65-39f9-496b-be1b-88338ca5e330" (UID: "c70ebf65-39f9-496b-be1b-88338ca5e330"). InnerVolumeSpecName "kube-api-access-56msl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:19:53.647787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.647742    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56msl\" (UniqueName: \"kubernetes.io/projected/c70ebf65-39f9-496b-be1b-88338ca5e330-kube-api-access-56msl\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:53.647787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:53.647782    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ebf65-39f9-496b-be1b-88338ca5e330-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:19:54.241623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:54.241545    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"
Oct 31 00:19:54.241776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:54.241542    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc" event={"ID":"c70ebf65-39f9-496b-be1b-88338ca5e330","Type":"ContainerDied","Data":"cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31"}
Oct 31 00:19:54.241776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:54.241682    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2fe106dfc7b1ee3c05f377d25e6621b9260de9bb02b038314d0dd543dbeb31"
Oct 31 00:19:56.177667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:56.177634    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:19:56.178095 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:19:56.177711    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:20:01.474192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:01.474151    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:20:05.464250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.464204    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:20:05.464938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.464918    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c70ebf65-39f9-496b-be1b-88338ca5e330" containerName="revoke-keys"
Oct 31 00:20:05.464938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.464937    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70ebf65-39f9-496b-be1b-88338ca5e330" containerName="revoke-keys"
Oct 31 00:20:05.465076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.465054    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70ebf65-39f9-496b-be1b-88338ca5e330" containerName="revoke-keys"
Oct 31 00:20:05.469335 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.469309    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.504171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.504133    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.504171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.504178    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.504450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.504291    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgq8\" (UniqueName: \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.504450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.504338    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.575653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.570673    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ae3cd0-serving-cert\""
Oct 31 00:20:05.575653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.574831    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:20:05.605210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.605172    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.605411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.605314    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.605411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.605344    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.605553 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.605435    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgq8\" (UniqueName: \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.607264 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:05.607234    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-serving-cert: secret "maas-api-e2e-api-a-ae3cd0-serving-cert" not found
Oct 31 00:20:05.607423 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:05.607323    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls podName:b254d093-f40a-4d3d-ab73-7e3c4694a5cd nodeName:}" failed. No retries permitted until 2026-10-31 00:20:06.107300507 +0000 UTC m=+2258.454284398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls") pod "maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" (UID: "b254d093-f40a-4d3d-ab73-7e3c4694a5cd") : secret "maas-api-e2e-api-a-ae3cd0-serving-cert" not found
Oct 31 00:20:05.608230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.608202    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.609249 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.609223    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:05.670742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:05.670705    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgq8\" (UniqueName: \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:06.112701 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.112591    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:06.115167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.115143    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") pod \"maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:06.386236 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.386137    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:06.474717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.474674    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" podUID="e52733d7-dd08-435a-a1f7-7581e0657f89" containerName="payload-processing" containerID="cri-o://40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434" gracePeriod=25
Oct 31 00:20:06.747962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.747930    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:20:06.762668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.762642    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:20:06.765021 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:06.764981    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb254d093_f40a_4d3d_ab73_7e3c4694a5cd.slice/crio-515241df376288aec8d92357c12746bd9a71874684e895298da227e674bf3eca WatchSource:0}: Error finding container 515241df376288aec8d92357c12746bd9a71874684e895298da227e674bf3eca: Status 404 returned error can't find the container with id 515241df376288aec8d92357c12746bd9a71874684e895298da227e674bf3eca
Oct 31 00:20:06.823463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.823430    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\") pod \"e52733d7-dd08-435a-a1f7-7581e0657f89\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") "
Oct 31 00:20:06.823691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.823514    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\" (UniqueName: \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\") pod \"e52733d7-dd08-435a-a1f7-7581e0657f89\" (UID: \"e52733d7-dd08-435a-a1f7-7581e0657f89\") "
Oct 31 00:20:06.823887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.823861    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume" pod "e52733d7-dd08-435a-a1f7-7581e0657f89" (UID: "e52733d7-dd08-435a-a1f7-7581e0657f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:06.824221 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.824201    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e52733d7-dd08-435a-a1f7-7581e0657f89-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:06.825783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.825759    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s" pod "e52733d7-dd08-435a-a1f7-7581e0657f89" (UID: "e52733d7-dd08-435a-a1f7-7581e0657f89"). InnerVolumeSpecName "kube-api-access-hjz8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:06.926892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:06.926803    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjz8s\" (UniqueName: \"kubernetes.io/projected/e52733d7-dd08-435a-a1f7-7581e0657f89-kube-api-access-hjz8s\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:07.332813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.332774    6033 generic.go:356] "Generic (PLEG): container finished" podID="e52733d7-dd08-435a-a1f7-7581e0657f89" containerID="40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434" exitCode=0
Oct 31 00:20:07.333022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.332826    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"
Oct 31 00:20:07.333022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.332851    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" event={"ID":"e52733d7-dd08-435a-a1f7-7581e0657f89","Type":"ContainerDied","Data":"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"}
Oct 31 00:20:07.333022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.332907    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859" event={"ID":"e52733d7-dd08-435a-a1f7-7581e0657f89","Type":"ContainerDied","Data":"c292dbbb120549482e7425b501dd96609f5f52545993277203d821a3780a211f"}
Oct 31 00:20:07.333022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.332935    6033 scope.go:122] "RemoveContainer" containerID="40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"
Oct 31 00:20:07.334567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.334538    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" event={"ID":"b254d093-f40a-4d3d-ab73-7e3c4694a5cd","Type":"ContainerStarted","Data":"515241df376288aec8d92357c12746bd9a71874684e895298da227e674bf3eca"}
Oct 31 00:20:07.343823 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.343800    6033 scope.go:122] "RemoveContainer" containerID="40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"
Oct 31 00:20:07.344156 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:07.344137    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434\": container with ID starting with 40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434 not found: ID does not exist" containerID="40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"
Oct 31 00:20:07.344301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.344166    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434"} err="failed to get container status \"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434\": rpc error: code = NotFound desc = could not find container \"40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434\": container with ID starting with 40159e1ad87185370420c5ee5e72380cc4fdd269137839719783135ee04ff434 not found: ID does not exist"
Oct 31 00:20:07.502331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.502291    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:20:07.547985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.547951    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-f7b8d8-6b866d558b-hq859"]
Oct 31 00:20:07.830240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:07.830204    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52733d7-dd08-435a-a1f7-7581e0657f89" path="/var/lib/kubelet/pods/e52733d7-dd08-435a-a1f7-7581e0657f89/volumes"
Oct 31 00:20:08.347343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:08.347306    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" event={"ID":"b254d093-f40a-4d3d-ab73-7e3c4694a5cd","Type":"ContainerStarted","Data":"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"}
Oct 31 00:20:08.347531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:08.347505    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:08.460580 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:08.460511    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" podStartSLOduration=2.579724145 podStartE2EDuration="3.46049565s" totalImagesPullingTime="880.771505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:08.459241265 +0000 UTC m=+2260.806225164" watchObservedRunningTime="2026-10-31 00:20:08.46049565 +0000 UTC m=+2260.807479545"
Oct 31 00:20:14.362691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:14.362654    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:20:16.972014 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:16.971973    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:20:26.541866 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:26.541825    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:20:28.653422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:28.653383    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:20:28.653951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:28.653644    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" podUID="ec19024f-267e-48a7-9589-960c9892d889" containerName="maas-api" containerID="cri-o://df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023" gracePeriod=30
Oct 31 00:20:29.156862 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.019637    6033 scope.go:122] "RemoveContainer" containerID="05f5338fce009191eb7d8a5d9d4125797c69fcdd8c55ea48856d5ac4425df871"
Oct 31 00:20:29.257930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.257897    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:20:29.299216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.299182    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\") pod \"ec19024f-267e-48a7-9589-960c9892d889\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") "
Oct 31 00:20:29.299357 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.299307    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") pod \"ec19024f-267e-48a7-9589-960c9892d889\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") "
Oct 31 00:20:29.299450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.299431    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\" (UniqueName: \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\") pod \"ec19024f-267e-48a7-9589-960c9892d889\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") "
Oct 31 00:20:29.299531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.299515    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\") pod \"ec19024f-267e-48a7-9589-960c9892d889\" (UID: \"ec19024f-267e-48a7-9589-960c9892d889\") "
Oct 31 00:20:29.302479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.302443    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca" pod "ec19024f-267e-48a7-9589-960c9892d889" (UID: "ec19024f-267e-48a7-9589-960c9892d889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:29.303979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.303946    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs" pod "ec19024f-267e-48a7-9589-960c9892d889" (UID: "ec19024f-267e-48a7-9589-960c9892d889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:29.307228 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.307177    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls" pod "ec19024f-267e-48a7-9589-960c9892d889" (UID: "ec19024f-267e-48a7-9589-960c9892d889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:29.307228 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.307186    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8" pod "ec19024f-267e-48a7-9589-960c9892d889" (UID: "ec19024f-267e-48a7-9589-960c9892d889"). InnerVolumeSpecName "kube-api-access-mr8s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:29.401891 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.401836    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8s8\" (UniqueName: \"kubernetes.io/projected/ec19024f-267e-48a7-9589-960c9892d889-kube-api-access-mr8s8\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:29.401891 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.401884    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:29.402228 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.401900    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec19024f-267e-48a7-9589-960c9892d889-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:29.402228 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.401915    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec19024f-267e-48a7-9589-960c9892d889-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:29.516301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.516263    6033 generic.go:356] "Generic (PLEG): container finished" podID="ec19024f-267e-48a7-9589-960c9892d889" containerID="df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023" exitCode=0
Oct 31 00:20:29.516512 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.516310    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" event={"ID":"ec19024f-267e-48a7-9589-960c9892d889","Type":"ContainerDied","Data":"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"}
Oct 31 00:20:29.516512 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.516391    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz" event={"ID":"ec19024f-267e-48a7-9589-960c9892d889","Type":"ContainerDied","Data":"13c91160fc44926adcee46486761feb749e71bb7c58ac8291ee0064cfd3d0d6d"}
Oct 31 00:20:29.516512 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.516423    6033 scope.go:122] "RemoveContainer" containerID="df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"
Oct 31 00:20:29.516697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.516581    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"
Oct 31 00:20:29.529830 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.529798    6033 scope.go:122] "RemoveContainer" containerID="df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"
Oct 31 00:20:29.530274 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:29.530250    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023\": container with ID starting with df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023 not found: ID does not exist" containerID="df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"
Oct 31 00:20:29.530359 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.530286    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023"} err="failed to get container status \"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023\": rpc error: code = NotFound desc = could not find container \"df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023\": container with ID starting with df0afb6dffee6875027dfb5d79924ff77555e500d4b2c27d61c6b23752769023 not found: ID does not exist"
Oct 31 00:20:29.674003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.673969    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:20:29.678829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.678799    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-f7b8d8-86b76d959f-q9gvz"]
Oct 31 00:20:29.833716 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:29.833621    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec19024f-267e-48a7-9589-960c9892d889" path="/var/lib/kubelet/pods/ec19024f-267e-48a7-9589-960c9892d889/volumes"
Oct 31 00:20:30.943057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:30.942994    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"]
Oct 31 00:20:30.989646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:30.989604    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f7b8d8-f76pc"]
Oct 31 00:20:31.830003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:31.829966    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70ebf65-39f9-496b-be1b-88338ca5e330" path="/var/lib/kubelet/pods/c70ebf65-39f9-496b-be1b-88338ca5e330/volumes"
Oct 31 00:20:35.232536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.232490    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:20:35.233517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233491    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ec19024f-267e-48a7-9589-960c9892d889" containerName="maas-api"
Oct 31 00:20:35.233590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233520    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec19024f-267e-48a7-9589-960c9892d889" containerName="maas-api"
Oct 31 00:20:35.233590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233558    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e52733d7-dd08-435a-a1f7-7581e0657f89" containerName="payload-processing"
Oct 31 00:20:35.233590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233567    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52733d7-dd08-435a-a1f7-7581e0657f89" containerName="payload-processing"
Oct 31 00:20:35.233739 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233725    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52733d7-dd08-435a-a1f7-7581e0657f89" containerName="payload-processing"
Oct 31 00:20:35.233787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.233753    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec19024f-267e-48a7-9589-960c9892d889" containerName="maas-api"
Oct 31 00:20:35.240119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.240087    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.284154 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.284115    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.284353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.284240    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98kr2\" (UniqueName: \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.292984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.292937    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:20:35.345055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.337495    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-49837c-dockercfg-bbvrr\""
Oct 31 00:20:35.345055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.337649    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-49837c\""
Oct 31 00:20:35.392052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.388121    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.392052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.388219    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98kr2\" (UniqueName: \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.392052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.389548    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.397733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.397692    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:20:35.403865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.403829    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.447331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.447302    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98kr2\" (UniqueName: \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\") pod \"payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.453390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.453351    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:20:35.489707 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.489621    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxds\" (UniqueName: \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.489707 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.489686    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.556422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.556376    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:35.590998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.590956    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxds\" (UniqueName: \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.591242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.591022    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.591756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.591732    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.649449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.649410    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxds\" (UniqueName: \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\") pod \"payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.724262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.724214    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:35.944428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:35.935641    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:20:35.944428 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:35.940385    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a14481_121f_4f88_8305_050e25db4f56.slice/crio-35a67d3a2e2fc3585145c30c4deaf7151df42ff258a69f96abe8f866db4e106e WatchSource:0}: Error finding container 35a67d3a2e2fc3585145c30c4deaf7151df42ff258a69f96abe8f866db4e106e: Status 404 returned error can't find the container with id 35a67d3a2e2fc3585145c30c4deaf7151df42ff258a69f96abe8f866db4e106e
Oct 31 00:20:36.112959 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.112931    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:20:36.115327 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:36.115283    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa0b10b_2d81_44dd_bd1f_3e9eee121bfc.slice/crio-41cfd0af3410fd323416b0d0912535ba4ec89dd7034efe3ac3447447710e586b WatchSource:0}: Error finding container 41cfd0af3410fd323416b0d0912535ba4ec89dd7034efe3ac3447447710e586b: Status 404 returned error can't find the container with id 41cfd0af3410fd323416b0d0912535ba4ec89dd7034efe3ac3447447710e586b
Oct 31 00:20:36.567796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.567749    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" event={"ID":"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc","Type":"ContainerStarted","Data":"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"}
Oct 31 00:20:36.568309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.567810    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" event={"ID":"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc","Type":"ContainerStarted","Data":"41cfd0af3410fd323416b0d0912535ba4ec89dd7034efe3ac3447447710e586b"}
Oct 31 00:20:36.568309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.567839    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:36.569624 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.569593    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" event={"ID":"c4a14481-121f-4f88-8305-050e25db4f56","Type":"ContainerStarted","Data":"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"}
Oct 31 00:20:36.569767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.569641    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" event={"ID":"c4a14481-121f-4f88-8305-050e25db4f56","Type":"ContainerStarted","Data":"35a67d3a2e2fc3585145c30c4deaf7151df42ff258a69f96abe8f866db4e106e"}
Oct 31 00:20:36.569767 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.569661    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:36.680266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.680207    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" podStartSLOduration=1.680189031 podStartE2EDuration="1.680189031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:36.677537915 +0000 UTC m=+2289.024521822" watchObservedRunningTime="2026-10-31 00:20:36.680189031 +0000 UTC m=+2289.027172928"
Oct 31 00:20:36.787773 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.787710    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" podStartSLOduration=1.787692319 podStartE2EDuration="1.787692319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:36.786936764 +0000 UTC m=+2289.133920662" watchObservedRunningTime="2026-10-31 00:20:36.787692319 +0000 UTC m=+2289.134676218"
Oct 31 00:20:36.871601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.871517    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"]
Oct 31 00:20:36.875976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.875944    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:36.907118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.907074    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75vr\" (UniqueName: \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:36.907435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.907406    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:36.907588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.907486    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:36.926517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.926482    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:20:36.926930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.926605    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:20:36.929924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:36.929895    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"]
Oct 31 00:20:37.008858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.008811    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.009087 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.008912    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75vr\" (UniqueName: \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.009162 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.009143    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.010594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.010561    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.013762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.013010    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.069190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.069149    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75vr\" (UniqueName: \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\") pod \"maas-api-revoke-sub-ff797a69-k4fz9\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.190902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.190808    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:37.570144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:37.570118    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"]
Oct 31 00:20:37.571790 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:37.571755    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44320c59_e6db_4362_ad41_83eafcaca1bb.slice/crio-1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b WatchSource:0}: Error finding container 1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b: Status 404 returned error can't find the container with id 1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b
Oct 31 00:20:38.588892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:38.588860    6033 generic.go:356] "Generic (PLEG): container finished" podID="44320c59-e6db-4362-ad41-83eafcaca1bb" containerID="3966a13fc4c9831350b6e1b9975affd835f58c9d1c741637ac849196a187fe59" exitCode=0
Oct 31 00:20:38.589411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:38.588917    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9" event={"ID":"44320c59-e6db-4362-ad41-83eafcaca1bb","Type":"ContainerDied","Data":"3966a13fc4c9831350b6e1b9975affd835f58c9d1c741637ac849196a187fe59"}
Oct 31 00:20:38.589411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:38.588971    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9" event={"ID":"44320c59-e6db-4362-ad41-83eafcaca1bb","Type":"ContainerStarted","Data":"1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b"}
Oct 31 00:20:38.630879 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:38.630837    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:20:38.631131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:38.631093    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy" containerID="cri-o://c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae" gracePeriod=30
Oct 31 00:20:39.488505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.488467    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"]
Oct 31 00:20:39.492695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.492673    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.541870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.541831    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.542184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.541990    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.542282 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.542230    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rv5\" (UniqueName: \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.546068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.546021    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"]
Oct 31 00:20:39.643963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.643917    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.644433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.644065    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rv5\" (UniqueName: \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.644433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.644155    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.644888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.644863    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.647265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.647232    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.707975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.707946    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rv5\" (UniqueName: \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.763682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.763645    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:39.806615 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.806573    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:39.846131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.846090    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\" (UniqueName: \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\") pod \"44320c59-e6db-4362-ad41-83eafcaca1bb\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") "
Oct 31 00:20:39.846304 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.846190    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\") pod \"44320c59-e6db-4362-ad41-83eafcaca1bb\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") "
Oct 31 00:20:39.846366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.846304    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\") pod \"44320c59-e6db-4362-ad41-83eafcaca1bb\" (UID: \"44320c59-e6db-4362-ad41-83eafcaca1bb\") "
Oct 31 00:20:39.846682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.846639    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca" pod "44320c59-e6db-4362-ad41-83eafcaca1bb" (UID: "44320c59-e6db-4362-ad41-83eafcaca1bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:39.846893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.846874    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:39.848320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.848294    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca" pod "44320c59-e6db-4362-ad41-83eafcaca1bb" (UID: "44320c59-e6db-4362-ad41-83eafcaca1bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:39.848409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.848327    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr" pod "44320c59-e6db-4362-ad41-83eafcaca1bb" (UID: "44320c59-e6db-4362-ad41-83eafcaca1bb"). InnerVolumeSpecName "kube-api-access-v75vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:39.948045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.947990    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75vr\" (UniqueName: \"kubernetes.io/projected/44320c59-e6db-4362-ad41-83eafcaca1bb-kube-api-access-v75vr\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:39.948206 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:39.948056    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44320c59-e6db-4362-ad41-83eafcaca1bb-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:40.180997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.180964    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"]
Oct 31 00:20:40.182551 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:40.182509    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75547e48_b6dc_4394_a82a_9f60b9b076f4.slice/crio-79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506 WatchSource:0}: Error finding container 79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506: Status 404 returned error can't find the container with id 79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506
Oct 31 00:20:40.605075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.605002    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq" event={"ID":"75547e48-b6dc-4394-a82a-9f60b9b076f4","Type":"ContainerStarted","Data":"a45aa59c3a060754b9ab39a322d029ecb00cca20c701c0e6aba9c03486c9dcde"}
Oct 31 00:20:40.605271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.605085    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq" event={"ID":"75547e48-b6dc-4394-a82a-9f60b9b076f4","Type":"ContainerStarted","Data":"79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506"}
Oct 31 00:20:40.606703 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.606669    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9" event={"ID":"44320c59-e6db-4362-ad41-83eafcaca1bb","Type":"ContainerDied","Data":"1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b"}
Oct 31 00:20:40.606827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.606722    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a9a9605b1b764d2ef3fe9534085d2fbe8ce868ab7bca42726a58201fab21b8b"
Oct 31 00:20:40.606827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.606678    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"
Oct 31 00:20:40.716297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:40.716239    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq" podStartSLOduration=1.7162229450000002 podStartE2EDuration="1.716222945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:40.714751914 +0000 UTC m=+2293.061735826" watchObservedRunningTime="2026-10-31 00:20:40.716222945 +0000 UTC m=+2293.063206884"
Oct 31 00:20:41.616184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:41.616146    6033 generic.go:356] "Generic (PLEG): container finished" podID="75547e48-b6dc-4394-a82a-9f60b9b076f4" containerID="a45aa59c3a060754b9ab39a322d029ecb00cca20c701c0e6aba9c03486c9dcde" exitCode=0
Oct 31 00:20:41.616344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:41.616228    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq" event={"ID":"75547e48-b6dc-4394-a82a-9f60b9b076f4","Type":"ContainerDied","Data":"a45aa59c3a060754b9ab39a322d029ecb00cca20c701c0e6aba9c03486c9dcde"}
Oct 31 00:20:42.219342 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.219302    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:20:42.219943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.219929    6033 cpu_manager.go:414] "containerMap: removing container" podUID="44320c59-e6db-4362-ad41-83eafcaca1bb" containerName="revoke-keys"
Oct 31 00:20:42.219983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.219944    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44320c59-e6db-4362-ad41-83eafcaca1bb" containerName="revoke-keys"
Oct 31 00:20:42.220064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.220053    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="44320c59-e6db-4362-ad41-83eafcaca1bb" containerName="revoke-keys"
Oct 31 00:20:42.223668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.223642    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.278741 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.278702    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfjh\" (UniqueName: \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.279066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.279025    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.280191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.280122    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d687ff\""
Oct 31 00:20:42.281711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.281684    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:20:42.323456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.323414    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d687ff-dockercfg-xxphj\""
Oct 31 00:20:42.374067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.373987    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:20:42.379564 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.379533    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.380225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.380199    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfjh\" (UniqueName: \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.380351 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.380254    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.381156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.381123    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.430674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.430637    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:20:42.458137 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.458099    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfjh\" (UniqueName: \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\") pod \"payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.482006 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.481910    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.482206 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.482075    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7hfv\" (UniqueName: \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.537967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.537916    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:42.583561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.583523    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.583779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.583689    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7hfv\" (UniqueName: \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.584388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.584365    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.642555 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.642518    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7hfv\" (UniqueName: \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\") pod \"payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") " pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.709910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.709874    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:42.777430 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.777388    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:42.886485 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.886443    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\") pod \"75547e48-b6dc-4394-a82a-9f60b9b076f4\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") "
Oct 31 00:20:42.886721 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.886609    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\" (UniqueName: \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\") pod \"75547e48-b6dc-4394-a82a-9f60b9b076f4\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") "
Oct 31 00:20:42.886803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.886768    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\") pod \"75547e48-b6dc-4394-a82a-9f60b9b076f4\" (UID: \"75547e48-b6dc-4394-a82a-9f60b9b076f4\") "
Oct 31 00:20:42.887383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.887347    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca" pod "75547e48-b6dc-4394-a82a-9f60b9b076f4" (UID: "75547e48-b6dc-4394-a82a-9f60b9b076f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:42.889190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.889128    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca" pod "75547e48-b6dc-4394-a82a-9f60b9b076f4" (UID: "75547e48-b6dc-4394-a82a-9f60b9b076f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:42.890168 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.890135    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5" pod "75547e48-b6dc-4394-a82a-9f60b9b076f4" (UID: "75547e48-b6dc-4394-a82a-9f60b9b076f4"). InnerVolumeSpecName "kube-api-access-48rv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:42.914906 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.914866    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:20:42.919638 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:42.919602    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9783b1_d7b7_44a1_8bef_8a323435aee2.slice/crio-a42fb2daa2bb54376b4c769cd280e48b41597ea35a809463da9f3efbdcbfcc3e WatchSource:0}: Error finding container a42fb2daa2bb54376b4c769cd280e48b41597ea35a809463da9f3efbdcbfcc3e: Status 404 returned error can't find the container with id a42fb2daa2bb54376b4c769cd280e48b41597ea35a809463da9f3efbdcbfcc3e
Oct 31 00:20:42.988445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.988419    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rv5\" (UniqueName: \"kubernetes.io/projected/75547e48-b6dc-4394-a82a-9f60b9b076f4-kube-api-access-48rv5\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:42.988509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.988450    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:42.988509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:42.988460    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75547e48-b6dc-4394-a82a-9f60b9b076f4-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:43.099848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.099815    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:20:43.101342 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:43.101311    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad36225a_ea11_4d9a_8efd_8e917933bc5f.slice/crio-21c5671e6ab3d6eac5ff82e9f168920f628b6d7512f806a2278cb5530966c474 WatchSource:0}: Error finding container 21c5671e6ab3d6eac5ff82e9f168920f628b6d7512f806a2278cb5530966c474: Status 404 returned error can't find the container with id 21c5671e6ab3d6eac5ff82e9f168920f628b6d7512f806a2278cb5530966c474
Oct 31 00:20:43.633313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.633279    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"
Oct 31 00:20:43.633766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.633313    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq" event={"ID":"75547e48-b6dc-4394-a82a-9f60b9b076f4","Type":"ContainerDied","Data":"79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506"}
Oct 31 00:20:43.633766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.633362    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79fc3bbbc462e435e1b76955f5bd240191c781eadeea4304a5bbcfd5d06bb506"
Oct 31 00:20:43.635171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.635135    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" event={"ID":"0a9783b1-d7b7-44a1-8bef-8a323435aee2","Type":"ContainerStarted","Data":"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"}
Oct 31 00:20:43.635293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.635213    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:43.635293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.635241    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" event={"ID":"0a9783b1-d7b7-44a1-8bef-8a323435aee2","Type":"ContainerStarted","Data":"a42fb2daa2bb54376b4c769cd280e48b41597ea35a809463da9f3efbdcbfcc3e"}
Oct 31 00:20:43.637275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.637250    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" event={"ID":"ad36225a-ea11-4d9a-8efd-8e917933bc5f","Type":"ContainerStarted","Data":"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"}
Oct 31 00:20:43.637364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.637299    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" event={"ID":"ad36225a-ea11-4d9a-8efd-8e917933bc5f","Type":"ContainerStarted","Data":"21c5671e6ab3d6eac5ff82e9f168920f628b6d7512f806a2278cb5530966c474"}
Oct 31 00:20:43.637364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.637327    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:43.745870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.745755    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" podStartSLOduration=1.745741281 podStartE2EDuration="1.745741281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:43.744305488 +0000 UTC m=+2296.091289388" watchObservedRunningTime="2026-10-31 00:20:43.745741281 +0000 UTC m=+2296.092725179"
Oct 31 00:20:43.910492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.909418    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" podStartSLOduration=1.90937512 podStartE2EDuration="1.90937512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:43.904043156 +0000 UTC m=+2296.251027047" watchObservedRunningTime="2026-10-31 00:20:43.90937512 +0000 UTC m=+2296.256359016"
Oct 31 00:20:43.923276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:43.923246    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:20:44.003171 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003135    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003197    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003257    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003301    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003349    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003372    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003391    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003470    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003532 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003498    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\") pod \"10e5fd67-710d-46af-a46e-113e9177b836\" (UID: \"10e5fd67-710d-46af-a46e-113e9177b836\") "
Oct 31 00:20:44.003738 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003543    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:44.003738 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003657    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:44.003855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003817    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:44.003940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003918    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:44.003940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.003924    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:44.004143 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.004120    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.004259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.004148    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.004259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.004163    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e5fd67-710d-46af-a46e-113e9177b836-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.004259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.004177    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.004259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.004190    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.005748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.005716    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:44.005869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.005841    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:20:44.005993 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.005976    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:20:44.006048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.005999    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt" pod "10e5fd67-710d-46af-a46e-113e9177b836" (UID: "10e5fd67-710d-46af-a46e-113e9177b836"). InnerVolumeSpecName "kube-api-access-zl7dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:44.105592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.105550    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7dt\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-kube-api-access-zl7dt\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.105592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.105584    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e5fd67-710d-46af-a46e-113e9177b836-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.105592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.105600    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e5fd67-710d-46af-a46e-113e9177b836-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.105833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.105614    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e5fd67-710d-46af-a46e-113e9177b836-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:44.647540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.647506    6033 generic.go:356] "Generic (PLEG): container finished" podID="10e5fd67-710d-46af-a46e-113e9177b836" containerID="c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae" exitCode=0
Oct 31 00:20:44.647942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.647558    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"
Oct 31 00:20:44.647942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.647586    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerDied","Data":"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"}
Oct 31 00:20:44.647942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.647647    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw" event={"ID":"10e5fd67-710d-46af-a46e-113e9177b836","Type":"ContainerDied","Data":"fc8674e10877f18c466f76bf2eafe4b36f9ccc5b501de388a4984eb7bcf7d192"}
Oct 31 00:20:44.647942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.647677    6033 scope.go:122] "RemoveContainer" containerID="c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"
Oct 31 00:20:44.659698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.659672    6033 scope.go:122] "RemoveContainer" containerID="3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"
Oct 31 00:20:44.671169 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.671139    6033 scope.go:122] "RemoveContainer" containerID="c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"
Oct 31 00:20:44.671527 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:44.671503    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae\": container with ID starting with c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae not found: ID does not exist" containerID="c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"
Oct 31 00:20:44.671602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.671544    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae"} err="failed to get container status \"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae\": rpc error: code = NotFound desc = could not find container \"c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae\": container with ID starting with c434365010b0e61a7cd22b4b40a189c0d6eff31c4ef8f9eec607b91e27d632ae not found: ID does not exist"
Oct 31 00:20:44.671602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.671580    6033 scope.go:122] "RemoveContainer" containerID="3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"
Oct 31 00:20:44.671920 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:44.671888    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970\": container with ID starting with 3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970 not found: ID does not exist" containerID="3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"
Oct 31 00:20:44.671963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.671930    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970"} err="failed to get container status \"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970\": rpc error: code = NotFound desc = could not find container \"3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970\": container with ID starting with 3b6de029099181ba2d724c579559d860216630068beb8eecc5dee7ab654f0970 not found: ID does not exist"
Oct 31 00:20:44.852176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.852138    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:20:44.898711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:44.898616    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-f7b8d8-openshift-default-6b8ffbcc97-cccbw"]
Oct 31 00:20:45.468790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.468750    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:20:45.469534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469509    6033 cpu_manager.go:414] "containerMap: removing container" podUID="75547e48-b6dc-4394-a82a-9f60b9b076f4" containerName="revoke-keys"
Oct 31 00:20:45.469534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469532    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75547e48-b6dc-4394-a82a-9f60b9b076f4" containerName="revoke-keys"
Oct 31 00:20:45.469726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469552    6033 cpu_manager.go:414] "containerMap: removing container" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:45.469726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469559    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:45.469726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469575    6033 cpu_manager.go:414] "containerMap: removing container" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:45.469726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469583    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:45.469935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469736    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:45.469935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.469769    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="75547e48-b6dc-4394-a82a-9f60b9b076f4" containerName="revoke-keys"
Oct 31 00:20:45.475063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.475021    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.537354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.537313    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ddf61a-serving-cert\""
Oct 31 00:20:45.569562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.569511    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:20:45.622315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.622260    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.622488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.622421    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlgr\" (UniqueName: \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.622592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.622578    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.622630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.622611    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.724329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.724225    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlgr\" (UniqueName: \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.724796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.724358    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.724796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.724389    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.724796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.724437    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.724796 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:45.724521    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-serving-cert: secret "maas-api-e2e-inf-a-ddf61a-serving-cert" not found
Oct 31 00:20:45.724796 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:45.724597    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls podName:1d0b0037-1655-4a52-a525-b72cb0a503dd nodeName:}" failed. No retries permitted until 2026-10-31 00:20:46.224578229 +0000 UTC m=+2298.571562122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls") pod "maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd") : secret "maas-api-e2e-inf-a-ddf61a-serving-cert" not found
Oct 31 00:20:45.725301 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.725275    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.727019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.726996    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.785204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.785156    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlgr\" (UniqueName: \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:45.831367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:45.831328    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e5fd67-710d-46af-a46e-113e9177b836" path="/var/lib/kubelet/pods/10e5fd67-710d-46af-a46e-113e9177b836/volumes"
Oct 31 00:20:46.231927 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:46.231888    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:46.232104 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:46.232073    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-serving-cert: secret "maas-api-e2e-inf-a-ddf61a-serving-cert" not found
Oct 31 00:20:46.232170 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:46.232137    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls podName:1d0b0037-1655-4a52-a525-b72cb0a503dd nodeName:}" failed. No retries permitted until 2026-10-31 00:20:47.232122061 +0000 UTC m=+2299.579105937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls") pod "maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd") : secret "maas-api-e2e-inf-a-ddf61a-serving-cert" not found
Oct 31 00:20:46.972661 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:20:46.972620    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:20:47.244554 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.244456    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:47.247239 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.247210    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:47.291466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.291428    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:47.580826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.580796    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:20:47.581021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.580865    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:20:47.689072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.687708    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:20:47.697378 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:47.697233    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b0037_1655_4a52_a525_b72cb0a503dd.slice/crio-14487627886ca4cd3c2ae740ae1a2034b2f4107be54498c495494560fd07aef2 WatchSource:0}: Error finding container 14487627886ca4cd3c2ae740ae1a2034b2f4107be54498c495494560fd07aef2: Status 404 returned error can't find the container with id 14487627886ca4cd3c2ae740ae1a2034b2f4107be54498c495494560fd07aef2
Oct 31 00:20:47.700076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:47.700048    6033 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:20:48.114467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.114422    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"]
Oct 31 00:20:48.115187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.115169    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e5fd67-710d-46af-a46e-113e9177b836" containerName="istio-proxy"
Oct 31 00:20:48.120151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.120121    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.171561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.171520    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:20:48.171779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.171664    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:20:48.222613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.222562    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"]
Oct 31 00:20:48.262853 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.262806    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.263108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.262879    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.263108 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.263084    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccwnd\" (UniqueName: \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.364806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.364698    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.364976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.364868    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.365083 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.364988    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccwnd\" (UniqueName: \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.365582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.365556    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.367480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.367454    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.427019 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.426977    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccwnd\" (UniqueName: \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\") pod \"maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.438887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.438852    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:48.687440 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.687400    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" event={"ID":"1d0b0037-1655-4a52-a525-b72cb0a503dd","Type":"ContainerStarted","Data":"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"}
Oct 31 00:20:48.687679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.687468    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" event={"ID":"1d0b0037-1655-4a52-a525-b72cb0a503dd","Type":"ContainerStarted","Data":"14487627886ca4cd3c2ae740ae1a2034b2f4107be54498c495494560fd07aef2"}
Oct 31 00:20:48.687679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.687493    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:48.801985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.801910    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" podStartSLOduration=2.929047631 podStartE2EDuration="3.801894564s" totalImagesPullingTime="872.846933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:48.801252197 +0000 UTC m=+2301.148236096" watchObservedRunningTime="2026-10-31 00:20:48.801894564 +0000 UTC m=+2301.148878508"
Oct 31 00:20:48.817046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:48.816998    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"]
Oct 31 00:20:48.819647 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:48.819617    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4503fed0_6380_49f1_a6b2_90742f139c7e.slice/crio-5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb WatchSource:0}: Error finding container 5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb: Status 404 returned error can't find the container with id 5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb
Oct 31 00:20:49.697402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:49.697366    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc" event={"ID":"4503fed0-6380-49f1-a6b2-90742f139c7e","Type":"ContainerStarted","Data":"56f434d2beea260f51feca173bf0cccc11a932715d26df468ff72b9bf2c21efc"}
Oct 31 00:20:49.698278 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:49.697423    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc" event={"ID":"4503fed0-6380-49f1-a6b2-90742f139c7e","Type":"ContainerStarted","Data":"5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb"}
Oct 31 00:20:49.809811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:49.809741    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc" podStartSLOduration=1.809720162 podStartE2EDuration="1.809720162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:49.808857241 +0000 UTC m=+2302.155841143" watchObservedRunningTime="2026-10-31 00:20:49.809720162 +0000 UTC m=+2302.156704060"
Oct 31 00:20:50.708641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:50.708596    6033 generic.go:356] "Generic (PLEG): container finished" podID="4503fed0-6380-49f1-a6b2-90742f139c7e" containerID="56f434d2beea260f51feca173bf0cccc11a932715d26df468ff72b9bf2c21efc" exitCode=0
Oct 31 00:20:50.709227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:50.708633    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc" event={"ID":"4503fed0-6380-49f1-a6b2-90742f139c7e","Type":"ContainerDied","Data":"56f434d2beea260f51feca173bf0cccc11a932715d26df468ff72b9bf2c21efc"}
Oct 31 00:20:51.880504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:51.880475    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:52.012769 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.012734    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\" (UniqueName: \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\") pod \"4503fed0-6380-49f1-a6b2-90742f139c7e\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") "
Oct 31 00:20:52.012974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.012909    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\") pod \"4503fed0-6380-49f1-a6b2-90742f139c7e\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") "
Oct 31 00:20:52.013065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.012982    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\") pod \"4503fed0-6380-49f1-a6b2-90742f139c7e\" (UID: \"4503fed0-6380-49f1-a6b2-90742f139c7e\") "
Oct 31 00:20:52.013382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.013352    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca" pod "4503fed0-6380-49f1-a6b2-90742f139c7e" (UID: "4503fed0-6380-49f1-a6b2-90742f139c7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:20:52.013572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.013548    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:52.015091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.015063    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca" pod "4503fed0-6380-49f1-a6b2-90742f139c7e" (UID: "4503fed0-6380-49f1-a6b2-90742f139c7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:20:52.015197 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.015176    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd" pod "4503fed0-6380-49f1-a6b2-90742f139c7e" (UID: "4503fed0-6380-49f1-a6b2-90742f139c7e"). InnerVolumeSpecName "kube-api-access-ccwnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:20:52.114382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.114345    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4503fed0-6380-49f1-a6b2-90742f139c7e-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:52.114382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.114379    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccwnd\" (UniqueName: \"kubernetes.io/projected/4503fed0-6380-49f1-a6b2-90742f139c7e-kube-api-access-ccwnd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:20:52.574096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.574056    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 31 00:20:52.723811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.723777    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"
Oct 31 00:20:52.723811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.723803    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc" event={"ID":"4503fed0-6380-49f1-a6b2-90742f139c7e","Type":"ContainerDied","Data":"5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb"}
Oct 31 00:20:52.724071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:52.723863    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5120149c95ca69284126e0b61136c7ce44227a16d1f6d02310af716b08c8f7eb"
Oct 31 00:20:54.515433 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.515390    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:20:54.517159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.517110    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4503fed0-6380-49f1-a6b2-90742f139c7e" containerName="revoke-keys"
Oct 31 00:20:54.517429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.517408    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4503fed0-6380-49f1-a6b2-90742f139c7e" containerName="revoke-keys"
Oct 31 00:20:54.518566 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.518500    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4503fed0-6380-49f1-a6b2-90742f139c7e" containerName="revoke-keys"
Oct 31 00:20:54.525548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.525519    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.548816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.548777    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.548944    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.548982    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549012    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549145    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549157    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:20:54.549330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549236    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549276    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549317    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.549500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.549434    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.553788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.553759    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.650965    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651106    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651134    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651165    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651197    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651239    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651269    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqgx\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651297    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651292    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651369    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651416    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651457    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651496    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651503    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651532    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651612    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.651679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651681    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.652210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651729    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.652210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651767    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.652210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.651807    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.652530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.652504    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.652631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.652575    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.652706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.652662    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:20:54.652870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.652844    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.653397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.653370    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.654889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.654852    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.655318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.655270    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.703740 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.703707    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:20:54.711288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.711241    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.712337 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.712312    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-gtzrp\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.747684 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.747647    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:20:54.752785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.752747    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.752972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.752795    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqgx\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.752972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.752831    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.752972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.752873    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753045    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753104    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753139    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753336 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753294    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753374    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753443    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753541 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753294    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.753631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.753604    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.754201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.754175    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.754498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.754476    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.756530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.756508    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.756681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.756664    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.814941 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.814818    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.824409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.815380    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqgx\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\") pod \"maas-default-gateway-openshift-default-8f6f87f59-nn2nk\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:54.831690 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.831641    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:20:54.852084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.845763    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:54.870632 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:54.870476    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:55.260970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.260935    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:20:55.262725 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:55.262673    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c932b_2bbd_4454_92d0_dcddf8a4ddea.slice/crio-972d963b7511c90cb0e1e78636d6e7d5afbedb4f522f86f333699539d74885ba WatchSource:0}: Error finding container 972d963b7511c90cb0e1e78636d6e7d5afbedb4f522f86f333699539d74885ba: Status 404 returned error can't find the container with id 972d963b7511c90cb0e1e78636d6e7d5afbedb4f522f86f333699539d74885ba
Oct 31 00:20:55.265939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.265889    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.266132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.265988    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.266132 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.266074    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.288743 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.288713    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:20:55.291456 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:20:55.291429    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5d6279_2168_4c9d_907e_287eb1965948.slice/crio-2be89070768e03a26b2db062278e8ba9dd1ae8ec218c0e9be3bfe831650c95f5 WatchSource:0}: Error finding container 2be89070768e03a26b2db062278e8ba9dd1ae8ec218c0e9be3bfe831650c95f5: Status 404 returned error can't find the container with id 2be89070768e03a26b2db062278e8ba9dd1ae8ec218c0e9be3bfe831650c95f5
Oct 31 00:20:55.294259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.294226    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.294322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.294294    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.294366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.294323    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:20:55.756997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.756880    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" event={"ID":"8a5d6279-2168-4c9d-907e-287eb1965948","Type":"ContainerStarted","Data":"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"}
Oct 31 00:20:55.756997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.756951    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" event={"ID":"8a5d6279-2168-4c9d-907e-287eb1965948","Type":"ContainerStarted","Data":"2be89070768e03a26b2db062278e8ba9dd1ae8ec218c0e9be3bfe831650c95f5"}
Oct 31 00:20:55.758952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.758913    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" event={"ID":"150c932b-2bbd-4454-92d0-dcddf8a4ddea","Type":"ContainerStarted","Data":"54afce903555db05b83f8bc2bf3640a034697cd719db9af61be4e08b3908531d"}
Oct 31 00:20:55.759174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.758968    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" event={"ID":"150c932b-2bbd-4454-92d0-dcddf8a4ddea","Type":"ContainerStarted","Data":"972d963b7511c90cb0e1e78636d6e7d5afbedb4f522f86f333699539d74885ba"}
Oct 31 00:20:55.846114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.846076    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:20:55.877055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.870933    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:20:55.879423 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:55.879325    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podStartSLOduration=1.8792864900000001 podStartE2EDuration="1.87928649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:55.873758476 +0000 UTC m=+2308.220742375" watchObservedRunningTime="2026-10-31 00:20:55.87928649 +0000 UTC m=+2308.226270391"
Oct 31 00:20:56.847668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:56.847629    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:20:56.848116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:56.847687    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:20:56.873203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:56.873156    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:20:56.873354 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:56.873223    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:20:57.846122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:57.846078    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:20:57.846407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:57.846143    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:20:57.872284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:57.872244    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:20:57.872661 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:57.872312    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:20:58.850091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:58.846507    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:20:58.850091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:58.846593    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:20:58.871780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:58.871741    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:20:58.871973 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:58.871804    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:20:59.772919 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.772856    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podStartSLOduration=5.77284145 podStartE2EDuration="5.77284145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:56.014264485 +0000 UTC m=+2308.361248421" watchObservedRunningTime="2026-10-31 00:20:59.77284145 +0000 UTC m=+2312.119825353"
Oct 31 00:20:59.775227 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.775200    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 31 00:20:59.846735 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.846688    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:20:59.846951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.846770    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:20:59.872226 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.872182    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:20:59.872426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:20:59.872237    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:21:00.845714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:00.845680    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:21:00.846139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:00.845736    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:21:00.871194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:00.871157    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:21:00.871485 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:00.871459    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:21:01.845881 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:01.845839    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:21:01.846387 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:01.845906    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:21:01.871700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:01.871663    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:21:01.871890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:01.871742    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:21:02.846022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:02.845972    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-gtzrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 31 00:21:02.846444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:02.846063    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 31 00:21:02.871689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:02.871652    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-nn2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 00:21:02.871882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:02.871707    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 00:21:04.345377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.345336    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:21:04.346591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.346557    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:21:04.347519 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.347494    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:21:04.350513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.350490    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:21:04.350812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.350783    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:21:04.351747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.351724    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:21:04.775821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:04.775771    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing" containerID="cri-o://ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b" gracePeriod=25
Oct 31 00:21:07.596573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.596522    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main" containerID="cri-o://daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14" gracePeriod=45
Oct 31 00:21:07.888287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.888252    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 31 00:21:07.908762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.908726    6033 generic.go:356] "Generic (PLEG): container finished" podID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerID="daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14" exitCode=0
Oct 31 00:21:07.908962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.908786    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"
Oct 31 00:21:07.908962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.908810    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" event={"ID":"c2627161-2e28-4343-a4c6-412d8b1c3a00","Type":"ContainerDied","Data":"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"}
Oct 31 00:21:07.908962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.908851    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z" event={"ID":"c2627161-2e28-4343-a4c6-412d8b1c3a00","Type":"ContainerDied","Data":"ce1cb01e831199dfac2478860cff7a69ddbb5a18b7b902b73866e6d12eb02bc5"}
Oct 31 00:21:07.908962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.908869    6033 scope.go:122] "RemoveContainer" containerID="daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"
Oct 31 00:21:07.922900 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.922876    6033 scope.go:122] "RemoveContainer" containerID="daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"
Oct 31 00:21:07.923272 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:07.923249    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14\": container with ID starting with daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14 not found: ID does not exist" containerID="daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"
Oct 31 00:21:07.923339 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.923284    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14"} err="failed to get container status \"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14\": rpc error: code = NotFound desc = could not find container \"daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14\": container with ID starting with daee9be13abf12601c027ead36b7d2992e09c16b69a569c60c3798ef2930cf14 not found: ID does not exist"
Oct 31 00:21:07.946476 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946439    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.946685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946535    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\" (UniqueName: \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.946685 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946616    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.946816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946724    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.946816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946769    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.946917 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946871    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\") pod \"c2627161-2e28-4343-a4c6-412d8b1c3a00\" (UID: \"c2627161-2e28-4343-a4c6-412d8b1c3a00\") "
Oct 31 00:21:07.947023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.946979    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:07.947142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.947100    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:07.947190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.947142    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:07.947631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.947610    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:07.947691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.947638    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:07.947691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.947657    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:07.948829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.948799    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:07.948950 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.948831    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:07.948950 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:07.948936    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf" pod "c2627161-2e28-4343-a4c6-412d8b1c3a00" (UID: "c2627161-2e28-4343-a4c6-412d8b1c3a00"). InnerVolumeSpecName "kube-api-access-7q7pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:08.049047 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.048991    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q7pf\" (UniqueName: \"kubernetes.io/projected/c2627161-2e28-4343-a4c6-412d8b1c3a00-kube-api-access-7q7pf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.049047 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.049040    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2627161-2e28-4343-a4c6-412d8b1c3a00-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.049047 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.049067    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2627161-2e28-4343-a4c6-412d8b1c3a00-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:08.324649 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.324609    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 31 00:21:08.332253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.332214    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fe01fa/simulator-w5-fe01fa-kserve-699959f59c-fxg9z"]
Oct 31 00:21:08.715427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.715385    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:21:08.802118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:08.802076    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:21:09.831242 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:09.831198    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" path="/var/lib/kubelet/pods/c2627161-2e28-4343-a4c6-412d8b1c3a00/volumes"
Oct 31 00:21:10.272891 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.272849    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"]
Oct 31 00:21:10.273923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.273898    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main"
Oct 31 00:21:10.274053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.273926    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main"
Oct 31 00:21:10.274214 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.274199    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2627161-2e28-4343-a4c6-412d8b1c3a00" containerName="main"
Oct 31 00:21:10.280286 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.280255    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.322073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.322011    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"]
Oct 31 00:21:10.328021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.327987    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.331463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.331430    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"]
Oct 31 00:21:10.363952 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.363911    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:21:10.364178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.364122    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:21:10.382616 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382574    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.382843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382641    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtsd\" (UniqueName: \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.382843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382697    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.382843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382712    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294nn\" (UniqueName: \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.382843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382822    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.383133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.382847    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.397001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.396740    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"]
Oct 31 00:21:10.484736 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.484680    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtsd\" (UniqueName: \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.484972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.484789    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.484972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.484816    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294nn\" (UniqueName: \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.485131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.484979    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.485131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.485070    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.485706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.485673    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.486631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.486585    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.486975 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.486906    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.488366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.488341    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.488366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.488348    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.547583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.547499    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-294nn\" (UniqueName: \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\") pod \"maas-api-revoke-sub-1957ba97-hg7n6\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:10.547583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.547497    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtsd\" (UniqueName: \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\") pod \"maas-api-revoke-sub-9f32150c-zl6rx\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.600793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.600748    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:10.656647 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:10.656614    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:11.011481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.011379    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"]
Oct 31 00:21:11.012241 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:21:11.012210    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965ae132_1d4b_421d_8c95_626cb485cfce.slice/crio-8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd WatchSource:0}: Error finding container 8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd: Status 404 returned error can't find the container with id 8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd
Oct 31 00:21:11.120024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.119985    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"]
Oct 31 00:21:11.953144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.953008    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6" event={"ID":"bb091328-a1ee-4d24-94c6-04abd70923a3","Type":"ContainerStarted","Data":"c25eab86f4b5090231a58f081ce6fc0c23bb6996094f9b499cacb1c4f3c0b227"}
Oct 31 00:21:11.953144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.953104    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6" event={"ID":"bb091328-a1ee-4d24-94c6-04abd70923a3","Type":"ContainerStarted","Data":"c4a6ed4777a40442aca671be2b88bb3dcfffaf396812b58c440b677120400f27"}
Oct 31 00:21:11.954851 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.954821    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx" event={"ID":"965ae132-1d4b-421d-8c95-626cb485cfce","Type":"ContainerStarted","Data":"89c060a968f291ffbab49cac43faadd72cb83f7d408e708b1fec226b1a9c290b"}
Oct 31 00:21:11.954986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:11.954879    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx" event={"ID":"965ae132-1d4b-421d-8c95-626cb485cfce","Type":"ContainerStarted","Data":"8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd"}
Oct 31 00:21:12.211665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.211270    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx" podStartSLOduration=2.211245203 podStartE2EDuration="2.211245203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:12.205463382 +0000 UTC m=+2324.552447282" watchObservedRunningTime="2026-10-31 00:21:12.211245203 +0000 UTC m=+2324.558229100"
Oct 31 00:21:12.212750 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.212693    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6" podStartSLOduration=2.212673382 podStartE2EDuration="2.212673382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:12.090364095 +0000 UTC m=+2324.437347998" watchObservedRunningTime="2026-10-31 00:21:12.212673382 +0000 UTC m=+2324.559657280"
Oct 31 00:21:12.963584 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.963548    6033 generic.go:356] "Generic (PLEG): container finished" podID="bb091328-a1ee-4d24-94c6-04abd70923a3" containerID="c25eab86f4b5090231a58f081ce6fc0c23bb6996094f9b499cacb1c4f3c0b227" exitCode=0
Oct 31 00:21:12.963799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.963609    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6" event={"ID":"bb091328-a1ee-4d24-94c6-04abd70923a3","Type":"ContainerDied","Data":"c25eab86f4b5090231a58f081ce6fc0c23bb6996094f9b499cacb1c4f3c0b227"}
Oct 31 00:21:12.965875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.965840    6033 generic.go:356] "Generic (PLEG): container finished" podID="965ae132-1d4b-421d-8c95-626cb485cfce" containerID="89c060a968f291ffbab49cac43faadd72cb83f7d408e708b1fec226b1a9c290b" exitCode=0
Oct 31 00:21:12.966063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:12.965917    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx" event={"ID":"965ae132-1d4b-421d-8c95-626cb485cfce","Type":"ContainerDied","Data":"89c060a968f291ffbab49cac43faadd72cb83f7d408e708b1fec226b1a9c290b"}
Oct 31 00:21:14.158942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.158906    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:14.163007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.162979    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:14.264588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264547    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\") pod \"965ae132-1d4b-421d-8c95-626cb485cfce\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") "
Oct 31 00:21:14.264588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264595    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\") pod \"965ae132-1d4b-421d-8c95-626cb485cfce\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") "
Oct 31 00:21:14.264865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264706    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\" (UniqueName: \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\") pod \"bb091328-a1ee-4d24-94c6-04abd70923a3\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") "
Oct 31 00:21:14.264865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264748    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\") pod \"bb091328-a1ee-4d24-94c6-04abd70923a3\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") "
Oct 31 00:21:14.264983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264873    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\") pod \"bb091328-a1ee-4d24-94c6-04abd70923a3\" (UID: \"bb091328-a1ee-4d24-94c6-04abd70923a3\") "
Oct 31 00:21:14.264983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.264932    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\" (UniqueName: \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\") pod \"965ae132-1d4b-421d-8c95-626cb485cfce\" (UID: \"965ae132-1d4b-421d-8c95-626cb485cfce\") "
Oct 31 00:21:14.265163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.265133    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca" pod "965ae132-1d4b-421d-8c95-626cb485cfce" (UID: "965ae132-1d4b-421d-8c95-626cb485cfce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:14.265236 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.265218    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca" pod "bb091328-a1ee-4d24-94c6-04abd70923a3" (UID: "bb091328-a1ee-4d24-94c6-04abd70923a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:14.265606 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.265581    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.265719 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.265610    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.267160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.267132    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca" pod "965ae132-1d4b-421d-8c95-626cb485cfce" (UID: "965ae132-1d4b-421d-8c95-626cb485cfce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:14.267274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.267170    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn" pod "bb091328-a1ee-4d24-94c6-04abd70923a3" (UID: "bb091328-a1ee-4d24-94c6-04abd70923a3"). InnerVolumeSpecName "kube-api-access-294nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:14.267274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.267186    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd" pod "965ae132-1d4b-421d-8c95-626cb485cfce" (UID: "965ae132-1d4b-421d-8c95-626cb485cfce"). InnerVolumeSpecName "kube-api-access-cgtsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:14.267274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.267256    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca" pod "bb091328-a1ee-4d24-94c6-04abd70923a3" (UID: "bb091328-a1ee-4d24-94c6-04abd70923a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:14.367145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.367111    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-294nn\" (UniqueName: \"kubernetes.io/projected/bb091328-a1ee-4d24-94c6-04abd70923a3-kube-api-access-294nn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.367145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.367143    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb091328-a1ee-4d24-94c6-04abd70923a3-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.367145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.367152    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtsd\" (UniqueName: \"kubernetes.io/projected/965ae132-1d4b-421d-8c95-626cb485cfce-kube-api-access-cgtsd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.367411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.367162    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/965ae132-1d4b-421d-8c95-626cb485cfce-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:14.991514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.991478    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"
Oct 31 00:21:14.991724 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.991514    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6" event={"ID":"bb091328-a1ee-4d24-94c6-04abd70923a3","Type":"ContainerDied","Data":"c4a6ed4777a40442aca671be2b88bb3dcfffaf396812b58c440b677120400f27"}
Oct 31 00:21:14.991785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.991718    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a6ed4777a40442aca671be2b88bb3dcfffaf396812b58c440b677120400f27"
Oct 31 00:21:14.993487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.993456    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"
Oct 31 00:21:14.993666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.993487    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx" event={"ID":"965ae132-1d4b-421d-8c95-626cb485cfce","Type":"ContainerDied","Data":"8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd"}
Oct 31 00:21:14.993666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:14.993532    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bce7d037294531edaca221f5d35ec7a0b835cb1bc1c93d4020afdabbd3ab0dd"
Oct 31 00:21:16.971229 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:16.971191    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:21:18.714896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:18.714853    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:21:25.053196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.053123    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 31 00:21:25.070688 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.070653    6033 generic.go:356] "Generic (PLEG): container finished" podID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerID="ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b" exitCode=0
Oct 31 00:21:25.070829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.070696    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"
Oct 31 00:21:25.070829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.070726    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" event={"ID":"c51be1a0-9642-4530-9fe7-4d0239ea5876","Type":"ContainerDied","Data":"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"}
Oct 31 00:21:25.070829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.070780    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l" event={"ID":"c51be1a0-9642-4530-9fe7-4d0239ea5876","Type":"ContainerDied","Data":"3671ed379100d2a82aa7d955da4cd9bb7f28004274eb85da9ecf0acec4f77c24"}
Oct 31 00:21:25.070829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.070803    6033 scope.go:122] "RemoveContainer" containerID="ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"
Oct 31 00:21:25.082881 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.082844    6033 scope.go:122] "RemoveContainer" containerID="ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"
Oct 31 00:21:25.083292 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:25.083264    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b\": container with ID starting with ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b not found: ID does not exist" containerID="ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"
Oct 31 00:21:25.083393 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.083307    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b"} err="failed to get container status \"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b\": rpc error: code = NotFound desc = could not find container \"ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b\": container with ID starting with ab1d0a742650399266a0b116560591483ed25c539fbc4f55216d4cf0f7c1295b not found: ID does not exist"
Oct 31 00:21:25.131816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.131779    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\" (UniqueName: \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\") pod \"c51be1a0-9642-4530-9fe7-4d0239ea5876\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") "
Oct 31 00:21:25.132013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.131879    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\") pod \"c51be1a0-9642-4530-9fe7-4d0239ea5876\" (UID: \"c51be1a0-9642-4530-9fe7-4d0239ea5876\") "
Oct 31 00:21:25.132306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.132284    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume" pod "c51be1a0-9642-4530-9fe7-4d0239ea5876" (UID: "c51be1a0-9642-4530-9fe7-4d0239ea5876"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:25.132510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.132486    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51be1a0-9642-4530-9fe7-4d0239ea5876-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:25.134319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.134288    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg" pod "c51be1a0-9642-4530-9fe7-4d0239ea5876" (UID: "c51be1a0-9642-4530-9fe7-4d0239ea5876"). InnerVolumeSpecName "kube-api-access-hl7pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:25.233650 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.233610    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl7pg\" (UniqueName: \"kubernetes.io/projected/c51be1a0-9642-4530-9fe7-4d0239ea5876-kube-api-access-hl7pg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:25.486559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.486521    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 31 00:21:25.490729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.490696    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fe01fa-576c754f8c-8dm7l"]
Oct 31 00:21:25.830319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:25.830280    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" path="/var/lib/kubelet/pods/c51be1a0-9642-4530-9fe7-4d0239ea5876/volumes"
Oct 31 00:21:26.593642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.593603    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:21:26.594288 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594271    6033 cpu_manager.go:414] "containerMap: removing container" podUID="965ae132-1d4b-421d-8c95-626cb485cfce" containerName="revoke-keys"
Oct 31 00:21:26.594349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594289    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965ae132-1d4b-421d-8c95-626cb485cfce" containerName="revoke-keys"
Oct 31 00:21:26.594349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594308    6033 cpu_manager.go:414] "containerMap: removing container" podUID="bb091328-a1ee-4d24-94c6-04abd70923a3" containerName="revoke-keys"
Oct 31 00:21:26.594349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594315    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb091328-a1ee-4d24-94c6-04abd70923a3" containerName="revoke-keys"
Oct 31 00:21:26.594349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594335    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing"
Oct 31 00:21:26.594349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594340    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing"
Oct 31 00:21:26.594506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594438    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="965ae132-1d4b-421d-8c95-626cb485cfce" containerName="revoke-keys"
Oct 31 00:21:26.594506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594447    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb091328-a1ee-4d24-94c6-04abd70923a3" containerName="revoke-keys"
Oct 31 00:21:26.594506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.594458    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51be1a0-9642-4530-9fe7-4d0239ea5876" containerName="payload-pre-processing"
Oct 31 00:21:26.599949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.599924    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.650250 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.650216    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1e5c33-dockercfg-v66cr\""
Oct 31 00:21:26.652855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.652828    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:21:26.656407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.656379    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1e5c33\""
Oct 31 00:21:26.656758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.656737    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsgf\" (UniqueName: \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.657061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.657019    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.758642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.758450    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.758642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.758573    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsgf\" (UniqueName: \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.759967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.759874    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.763260 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.763229    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:21:26.769306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.769266    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:26.818574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.818527    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsgf\" (UniqueName: \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\") pod \"payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.818778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.818594    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:21:26.860466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.860358    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:26.860665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.860483    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8v4\" (UniqueName: \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:26.915518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.915479    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:26.962328 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.962275    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8v4\" (UniqueName: \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:26.962575 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.962557    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:26.963365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:26.963335    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:27.026323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:27.026285    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8v4\" (UniqueName: \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\") pod \"payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") " pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:27.094135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:27.094102    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:27.292717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:27.292685    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:21:27.293967 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:21:27.293935    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4386868_b38b_4fe2_990a_8a77e857562f.slice/crio-752b0be1cf362056d3b28a4aa7709f666db5ca441d1f373f767be196ce09849e WatchSource:0}: Error finding container 752b0be1cf362056d3b28a4aa7709f666db5ca441d1f373f767be196ce09849e: Status 404 returned error can't find the container with id 752b0be1cf362056d3b28a4aa7709f666db5ca441d1f373f767be196ce09849e
Oct 31 00:21:27.468855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:27.468824    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:21:27.470056 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:21:27.469989    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc3fe15_87dc_4c09_baf9_0a4e41c12e39.slice/crio-9d450294a2adc039959600fc204b8d2a43a239e8b5e4cdcd3278907c65972b5f WatchSource:0}: Error finding container 9d450294a2adc039959600fc204b8d2a43a239e8b5e4cdcd3278907c65972b5f: Status 404 returned error can't find the container with id 9d450294a2adc039959600fc204b8d2a43a239e8b5e4cdcd3278907c65972b5f
Oct 31 00:21:28.096411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.096371    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" event={"ID":"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39","Type":"ContainerStarted","Data":"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"}
Oct 31 00:21:28.096858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.096461    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" event={"ID":"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39","Type":"ContainerStarted","Data":"9d450294a2adc039959600fc204b8d2a43a239e8b5e4cdcd3278907c65972b5f"}
Oct 31 00:21:28.096858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.096539    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:28.098323 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.098285    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" event={"ID":"c4386868-b38b-4fe2-990a-8a77e857562f","Type":"ContainerStarted","Data":"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"}
Oct 31 00:21:28.098508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.098341    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" event={"ID":"c4386868-b38b-4fe2-990a-8a77e857562f","Type":"ContainerStarted","Data":"752b0be1cf362056d3b28a4aa7709f666db5ca441d1f373f767be196ce09849e"}
Oct 31 00:21:28.098508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.098380    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:28.208092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.208015    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" podStartSLOduration=2.208001045 podStartE2EDuration="2.208001045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:28.205977729 +0000 UTC m=+2340.552961627" watchObservedRunningTime="2026-10-31 00:21:28.208001045 +0000 UTC m=+2340.554984943"
Oct 31 00:21:28.316516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:28.316460    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" podStartSLOduration=2.316447672 podStartE2EDuration="2.316447672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:28.315674924 +0000 UTC m=+2340.662658835" watchObservedRunningTime="2026-10-31 00:21:28.316447672 +0000 UTC m=+2340.663431601"
Oct 31 00:21:29.312511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:29.312471    6033 scope.go:122] "RemoveContainer" containerID="e1a73c874148699c5a9eb55256775e2c1f4bf40ae95e3f96b14b503cc4c61f22"
Oct 31 00:21:35.575082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.575044    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:21:35.582121 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.582082    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.631765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.631727    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-49837c-serving-cert\""
Oct 31 00:21:35.681706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.681664    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:21:35.690203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.690169    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.690360 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.690235    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dftd\" (UniqueName: \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.690426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.690401    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.690466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.690445    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.792133 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.792091    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.792337 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.792257    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dftd\" (UniqueName: \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.792400 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.792381    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.792443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.792425    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.792552 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:35.792525    6033 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-serving-cert: secret "maas-api-e2e-inf-b-49837c-serving-cert" not found
Oct 31 00:21:35.792620 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:35.792596    6033 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls podName:860cf407-971d-4195-b15e-72331da6d9c2 nodeName:}" failed. No retries permitted until 2026-10-31 00:21:36.292580342 +0000 UTC m=+2348.639564231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls") pod "maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" (UID: "860cf407-971d-4195-b15e-72331da6d9c2") : secret "maas-api-e2e-inf-b-49837c-serving-cert" not found
Oct 31 00:21:35.792827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.792800    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.794949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.794924    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:35.867884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:35.867796    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dftd\" (UniqueName: \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:36.299621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:36.299577    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:36.302156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:36.302123    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") pod \"maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:36.497293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:36.497255    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:36.874412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:36.874382    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:21:36.877367 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:21:36.877336    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860cf407_971d_4195_b15e_72331da6d9c2.slice/crio-2d54ed4158ee7d56722a0d9c1c94b30c8dd7bbf63c815fe016bf7a9b1b3bbfd8 WatchSource:0}: Error finding container 2d54ed4158ee7d56722a0d9c1c94b30c8dd7bbf63c815fe016bf7a9b1b3bbfd8: Status 404 returned error can't find the container with id 2d54ed4158ee7d56722a0d9c1c94b30c8dd7bbf63c815fe016bf7a9b1b3bbfd8
Oct 31 00:21:37.170398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:37.170311    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" event={"ID":"860cf407-971d-4195-b15e-72331da6d9c2","Type":"ContainerStarted","Data":"2d54ed4158ee7d56722a0d9c1c94b30c8dd7bbf63c815fe016bf7a9b1b3bbfd8"}
Oct 31 00:21:38.179455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:38.179357    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" event={"ID":"860cf407-971d-4195-b15e-72331da6d9c2","Type":"ContainerStarted","Data":"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"}
Oct 31 00:21:38.179455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:38.179424    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:38.293632 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:38.293565    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" podStartSLOduration=2.418975504 podStartE2EDuration="3.293551397s" totalImagesPullingTime="874.575893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:38.291153642 +0000 UTC m=+2350.638137565" watchObservedRunningTime="2026-10-31 00:21:38.293551397 +0000 UTC m=+2350.640535294"
Oct 31 00:21:39.108987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:39.108956    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:21:39.109208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:39.109056    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:21:42.385699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:42.385656    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:21:44.192550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:44.192518    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:21:46.974101 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:21:46.974016    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:21:47.264844 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:47.264795    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"]
Oct 31 00:21:47.311158 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:47.311117    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0d75be-4wqbq"]
Oct 31 00:21:47.830597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:47.830559    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75547e48-b6dc-4394-a82a-9f60b9b076f4" path="/var/lib/kubelet/pods/75547e48-b6dc-4394-a82a-9f60b9b076f4/volumes"
Oct 31 00:21:55.632395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.632352    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"]
Oct 31 00:21:55.637002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.636974    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.686068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.685961    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.686305 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.686215    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.686305 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.686273    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6bc\" (UniqueName: \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.692246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.692208    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"]
Oct 31 00:21:55.734622 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.734583    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:21:55.734622 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.734610    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:21:55.788006 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.787945    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.788277 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.788156    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.788277 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.788199    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6bc\" (UniqueName: \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.788953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.788929    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.790643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.790621    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.855880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.855841    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6bc\" (UniqueName: \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\") pod \"maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:55.955626 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:55.955531    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:56.332304 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:56.332270    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"]
Oct 31 00:21:56.334503 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:21:56.334452    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae69c5b_a6b4_4015_aa32_d7e2270f5e5e.slice/crio-9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d WatchSource:0}: Error finding container 9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d: Status 404 returned error can't find the container with id 9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d
Oct 31 00:21:57.319446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:57.319413    6033 generic.go:356] "Generic (PLEG): container finished" podID="fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" containerID="6eb51e2f9adeaa20419c1dc7a48d5f06157fb89c8a8a50fb076ff0647b8d2b2d" exitCode=0
Oct 31 00:21:57.319864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:57.319491    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg" event={"ID":"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e","Type":"ContainerDied","Data":"6eb51e2f9adeaa20419c1dc7a48d5f06157fb89c8a8a50fb076ff0647b8d2b2d"}
Oct 31 00:21:57.319864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:57.319575    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg" event={"ID":"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e","Type":"ContainerStarted","Data":"9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d"}
Oct 31 00:21:58.479613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.479581    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:58.524540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.524505    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\" (UniqueName: \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\") pod \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") "
Oct 31 00:21:58.524731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.524692    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\") pod \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") "
Oct 31 00:21:58.524873 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.524856    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\") pod \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\" (UID: \"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e\") "
Oct 31 00:21:58.525044 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.525010    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca" pod "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" (UID: "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:58.525284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.525267    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:58.526826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.526794    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc" pod "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" (UID: "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e"). InnerVolumeSpecName "kube-api-access-fw6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:58.526826 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.526812    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca" pod "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" (UID: "fae69c5b-a6b4-4015-aa32-d7e2270f5e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:21:58.627244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.627147    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6bc\" (UniqueName: \"kubernetes.io/projected/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-kube-api-access-fw6bc\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:58.627244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:58.627190    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:21:59.335388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:59.335348    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"
Oct 31 00:21:59.335594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:59.335390    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg" event={"ID":"fae69c5b-a6b4-4015-aa32-d7e2270f5e5e","Type":"ContainerDied","Data":"9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d"}
Oct 31 00:21:59.335594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:21:59.335447    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb17d31093818e830cb65102208b967b7e547f55c6c355f30e6cbda846f883d"
Oct 31 00:22:00.047397 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:00.047349    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 31 00:22:07.066597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:07.066558    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 31 00:22:11.415519 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.415483    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"]
Oct 31 00:22:11.416349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.416327    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" containerName="revoke-keys"
Oct 31 00:22:11.416349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.416349    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" containerName="revoke-keys"
Oct 31 00:22:11.416514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.416479    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" containerName="revoke-keys"
Oct 31 00:22:11.421788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.421759    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.455527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.455486    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9mh\" (UniqueName: \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.455744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.455662    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.455744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.455715    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.520652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.520609    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:22:11.520905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.520706    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:22:11.525300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.525273    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"]
Oct 31 00:22:11.527894 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.527869    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"]
Oct 31 00:22:11.532274 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.532257    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.557208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.557174    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9mh\" (UniqueName: \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.557392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.557276    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.557392 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.557311    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.558110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.558087    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.559798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.559774    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.571966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.571931    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"]
Oct 31 00:22:11.615661 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.615629    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9mh\" (UniqueName: \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\") pod \"maas-api-revoke-sub-6eed3c84-mzvhd\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.658798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.658758    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.658988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.658851    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrz8\" (UniqueName: \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.659151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.659121    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.737755 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.737670    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:11.761001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.760962    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.761202 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.761083    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrz8\" (UniqueName: \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.761275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.761251    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.761753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.761724    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.763854 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.763826    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.823075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.821812    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrz8\" (UniqueName: \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\") pod \"maas-api-revoke-sub-431ac0f9-scd55\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:11.847163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:11.847130    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:12.067144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:12.067071    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing" containerID="cri-o://1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3" gracePeriod=25
Oct 31 00:22:12.114970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:12.114941    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"]
Oct 31 00:22:12.116087 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:22:12.116054    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b150ef_b32d_4dcc_abb1_da37a970d023.slice/crio-e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14 WatchSource:0}: Error finding container e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14: Status 404 returned error can't find the container with id e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14
Oct 31 00:22:12.225434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:12.225390    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"]
Oct 31 00:22:12.227340 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:22:12.227311    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5451011c_f57a_4c3e_9d82_455788d97bec.slice/crio-dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16 WatchSource:0}: Error finding container dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16: Status 404 returned error can't find the container with id dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16
Oct 31 00:22:12.441824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:12.441725    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55" event={"ID":"5451011c-f57a-4c3e-9d82-455788d97bec","Type":"ContainerStarted","Data":"dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16"}
Oct 31 00:22:12.443275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:12.443244    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd" event={"ID":"49b150ef-b32d-4dcc-abb1-da37a970d023","Type":"ContainerStarted","Data":"e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14"}
Oct 31 00:22:13.452751 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.452666    6033 generic.go:356] "Generic (PLEG): container finished" podID="5451011c-f57a-4c3e-9d82-455788d97bec" containerID="63e6e063ed2aead17a9f85502ed264c6fc76c53e79a5de2d5aa4c2df36e601bb" exitCode=0
Oct 31 00:22:13.453225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.452753    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55" event={"ID":"5451011c-f57a-4c3e-9d82-455788d97bec","Type":"ContainerDied","Data":"63e6e063ed2aead17a9f85502ed264c6fc76c53e79a5de2d5aa4c2df36e601bb"}
Oct 31 00:22:13.454663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.454639    6033 generic.go:356] "Generic (PLEG): container finished" podID="49b150ef-b32d-4dcc-abb1-da37a970d023" containerID="e446e952aea98d12ec643d9a8331e68f2fe1f18106832f2a1ea3ad5f1f98839e" exitCode=0
Oct 31 00:22:13.454778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.454715    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd" event={"ID":"49b150ef-b32d-4dcc-abb1-da37a970d023","Type":"ContainerDied","Data":"e446e952aea98d12ec643d9a8331e68f2fe1f18106832f2a1ea3ad5f1f98839e"}
Oct 31 00:22:13.783467 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.783428    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:22:13.787905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.787878    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:13.839094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.839062    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-37abc0\""
Oct 31 00:22:13.839343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.839261    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-37abc0-dockercfg-hh62x\""
Oct 31 00:22:13.842604 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.842565    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:22:13.898066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.897925    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bs5\" (UniqueName: \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:13.898331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:13.898290    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.000101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.000014    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bs5\" (UniqueName: \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.000265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.000240    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.000940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.000920    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.058670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.058588    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bs5\" (UniqueName: \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\") pod \"payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.102430 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.102387    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:14.478346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.478305    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:22:14.485653 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:22:14.485608    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4087af_277c_4ee8_a262_73477f30a70b.slice/crio-ba7bf6f2691270ad9015a089e73179ab0e6479bbcc244073f28eb82fe1c49c9a WatchSource:0}: Error finding container ba7bf6f2691270ad9015a089e73179ab0e6479bbcc244073f28eb82fe1c49c9a: Status 404 returned error can't find the container with id ba7bf6f2691270ad9015a089e73179ab0e6479bbcc244073f28eb82fe1c49c9a
Oct 31 00:22:14.614052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.611165    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:14.640077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.640004    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:14.712841 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.712745    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\" (UniqueName: \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\") pod \"49b150ef-b32d-4dcc-abb1-da37a970d023\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") "
Oct 31 00:22:14.712841 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.712792    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\") pod \"5451011c-f57a-4c3e-9d82-455788d97bec\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") "
Oct 31 00:22:14.713103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.712922    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\") pod \"49b150ef-b32d-4dcc-abb1-da37a970d023\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") "
Oct 31 00:22:14.713103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.712950    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\") pod \"5451011c-f57a-4c3e-9d82-455788d97bec\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") "
Oct 31 00:22:14.713103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.712995    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\" (UniqueName: \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\") pod \"5451011c-f57a-4c3e-9d82-455788d97bec\" (UID: \"5451011c-f57a-4c3e-9d82-455788d97bec\") "
Oct 31 00:22:14.713256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.713213    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\") pod \"49b150ef-b32d-4dcc-abb1-da37a970d023\" (UID: \"49b150ef-b32d-4dcc-abb1-da37a970d023\") "
Oct 31 00:22:14.713410 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.713373    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca" pod "5451011c-f57a-4c3e-9d82-455788d97bec" (UID: "5451011c-f57a-4c3e-9d82-455788d97bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:14.713667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.713637    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca" pod "49b150ef-b32d-4dcc-abb1-da37a970d023" (UID: "49b150ef-b32d-4dcc-abb1-da37a970d023"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:14.713898 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.713872    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:14.713977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.713904    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:14.715181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.715153    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh" pod "49b150ef-b32d-4dcc-abb1-da37a970d023" (UID: "49b150ef-b32d-4dcc-abb1-da37a970d023"). InnerVolumeSpecName "kube-api-access-wp9mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:14.715181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.715167    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca" pod "5451011c-f57a-4c3e-9d82-455788d97bec" (UID: "5451011c-f57a-4c3e-9d82-455788d97bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:14.715310 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.715219    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8" pod "5451011c-f57a-4c3e-9d82-455788d97bec" (UID: "5451011c-f57a-4c3e-9d82-455788d97bec"). InnerVolumeSpecName "kube-api-access-hsrz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:14.715310 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.715220    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca" pod "49b150ef-b32d-4dcc-abb1-da37a970d023" (UID: "49b150ef-b32d-4dcc-abb1-da37a970d023"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:14.815291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.815241    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9mh\" (UniqueName: \"kubernetes.io/projected/49b150ef-b32d-4dcc-abb1-da37a970d023-kube-api-access-wp9mh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:14.815291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.815280    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5451011c-f57a-4c3e-9d82-455788d97bec-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:14.815291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.815294    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49b150ef-b32d-4dcc-abb1-da37a970d023-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:14.815291 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:14.815304    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrz8\" (UniqueName: \"kubernetes.io/projected/5451011c-f57a-4c3e-9d82-455788d97bec-kube-api-access-hsrz8\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.070055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.069940    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" containerID="cri-o://f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138" gracePeriod=45
Oct 31 00:22:15.117209 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.117158    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 31 00:22:15.362599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.362568    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:22:15.470730 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.470685    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55" event={"ID":"5451011c-f57a-4c3e-9d82-455788d97bec","Type":"ContainerDied","Data":"dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16"}
Oct 31 00:22:15.470925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.470789    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd5816dad60a80e2c5db76400e6db7986c92335861f0942a9501c56d2fe01d16"
Oct 31 00:22:15.470925 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.470707    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"
Oct 31 00:22:15.472299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.472277    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"
Oct 31 00:22:15.472439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.472302    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd" event={"ID":"49b150ef-b32d-4dcc-abb1-da37a970d023","Type":"ContainerDied","Data":"e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14"}
Oct 31 00:22:15.472439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.472351    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c710fd2fff940dc1d60e4496ba23697498f97c983ca1dce933a8e787a4aa14"
Oct 31 00:22:15.479970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.479928    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" event={"ID":"7f4087af-277c-4ee8-a262-73477f30a70b","Type":"ContainerStarted","Data":"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"}
Oct 31 00:22:15.480639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.479998    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" event={"ID":"7f4087af-277c-4ee8-a262-73477f30a70b","Type":"ContainerStarted","Data":"ba7bf6f2691270ad9015a089e73179ab0e6479bbcc244073f28eb82fe1c49c9a"}
Oct 31 00:22:15.482284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.481219    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:15.483821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.483793    6033 generic.go:356] "Generic (PLEG): container finished" podID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerID="f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138" exitCode=0
Oct 31 00:22:15.483970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.483906    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" event={"ID":"e8d8639e-c06b-49e2-82f0-166c0308f907","Type":"ContainerDied","Data":"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"}
Oct 31 00:22:15.483970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.483940    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx" event={"ID":"e8d8639e-c06b-49e2-82f0-166c0308f907","Type":"ContainerDied","Data":"96e0779bba69783c81762661f4140776f9cdd4a2af616d642ebb9cd5145a1c57"}
Oct 31 00:22:15.483970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.483962    6033 scope.go:122] "RemoveContainer" containerID="f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"
Oct 31 00:22:15.484256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.484213    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"
Oct 31 00:22:15.498540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.497045    6033 scope.go:122] "RemoveContainer" containerID="f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"
Oct 31 00:22:15.498540 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:15.497680    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138\": container with ID starting with f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138 not found: ID does not exist" containerID="f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"
Oct 31 00:22:15.498540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.497726    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138"} err="failed to get container status \"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138\": rpc error: code = NotFound desc = could not find container \"f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138\": container with ID starting with f48b052f2ddf4e2df96c299f01752e58d552a7c2642825d14c9953d97b4b2138 not found: ID does not exist"
Oct 31 00:22:15.523709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.523668    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\" (UniqueName: \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.523912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.523732    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.523912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.523870    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.525075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.523938    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.525075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.524015    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.525075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.524090    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\") pod \"e8d8639e-c06b-49e2-82f0-166c0308f907\" (UID: \"e8d8639e-c06b-49e2-82f0-166c0308f907\") "
Oct 31 00:22:15.525075 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.524661    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.525563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.525440    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.525563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.525460    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.526678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.525802    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.528984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.528939    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "kube-api-access-ftsq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:15.529184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.529161    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:15.529275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.529194    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs" pod "e8d8639e-c06b-49e2-82f0-166c0308f907" (UID: "e8d8639e-c06b-49e2-82f0-166c0308f907"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:22:15.596668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.596548    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" podStartSLOduration=2.596527434 podStartE2EDuration="2.596527434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:15.595426705 +0000 UTC m=+2387.942410605" watchObservedRunningTime="2026-10-31 00:22:15.596527434 +0000 UTC m=+2387.943511333"
Oct 31 00:22:15.626691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.626643    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsq9\" (UniqueName: \"kubernetes.io/projected/e8d8639e-c06b-49e2-82f0-166c0308f907-kube-api-access-ftsq9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.626691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.626685    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.626691 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.626695    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.626931 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.626704    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8d8639e-c06b-49e2-82f0-166c0308f907-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.626931 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.626714    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8d8639e-c06b-49e2-82f0-166c0308f907-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:15.902382 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.902282    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 31 00:22:15.910011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:15.909975    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-35a187/premium-w6-35a187-kserve-67d4df985c-bl9jx"]
Oct 31 00:22:16.706913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:16.706864    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:22:16.970866 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:16.970780    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:22:17.830557 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:17.830518    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" path="/var/lib/kubelet/pods/e8d8639e-c06b-49e2-82f0-166c0308f907/volumes"
Oct 31 00:22:23.849011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:23.848964    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"]
Oct 31 00:22:23.895832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:23.895793    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fe01fa-wlhxc"]
Oct 31 00:22:25.830771 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:25.830734    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4503fed0-6380-49f1-a6b2-90742f139c7e" path="/var/lib/kubelet/pods/4503fed0-6380-49f1-a6b2-90742f139c7e/volumes"
Oct 31 00:22:26.706749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:26.706699    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:22:27.506139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:27.506102    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:22:29.364422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:29.364385    6033 scope.go:122] "RemoveContainer" containerID="ca66440fcd772b211edfe4a6b1bb0acdce423648ef19e65f58dfa4ba09705edf"
Oct 31 00:22:29.375964 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:29.375943    6033 scope.go:122] "RemoveContainer" containerID="d5f1c5eee2576e105f26fa7e7340126e10734b612debf9a1c3472ce1e4974ea6"
Oct 31 00:22:31.394204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:31.394155    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 31 00:22:31.394734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:31.394420    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy" containerID="cri-o://150055cfa5a6a4b9d7e4866b480002a54f54f8d2830c3cf790392d2e8c952881" gracePeriod=30
Oct 31 00:22:32.350001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.349971    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 31 00:22:32.490103 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.490016    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\") pod \"7387d560-e431-4455-82ea-20cae348ed33\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") "
Oct 31 00:22:32.490547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.490233    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\" (UniqueName: \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\") pod \"7387d560-e431-4455-82ea-20cae348ed33\" (UID: \"7387d560-e431-4455-82ea-20cae348ed33\") "
Oct 31 00:22:32.490547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.490452    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume" pod "7387d560-e431-4455-82ea-20cae348ed33" (UID: "7387d560-e431-4455-82ea-20cae348ed33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:32.490720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.490701    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7387d560-e431-4455-82ea-20cae348ed33-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:32.492638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.492584    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng" pod "7387d560-e431-4455-82ea-20cae348ed33" (UID: "7387d560-e431-4455-82ea-20cae348ed33"). InnerVolumeSpecName "kube-api-access-r2bng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:32.592289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.592198    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bng\" (UniqueName: \"kubernetes.io/projected/7387d560-e431-4455-82ea-20cae348ed33-kube-api-access-r2bng\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:32.614698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.614655    6033 generic.go:356] "Generic (PLEG): container finished" podID="7387d560-e431-4455-82ea-20cae348ed33" containerID="1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3" exitCode=0
Oct 31 00:22:32.614877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.614722    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"
Oct 31 00:22:32.614877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.614723    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" event={"ID":"7387d560-e431-4455-82ea-20cae348ed33","Type":"ContainerDied","Data":"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"}
Oct 31 00:22:32.614970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.614885    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv" event={"ID":"7387d560-e431-4455-82ea-20cae348ed33","Type":"ContainerDied","Data":"d98947e2f670303c6db7288e36e1a0a64d34f2bc596a278137c3368856d2f2ee"}
Oct 31 00:22:32.614970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.614908    6033 scope.go:122] "RemoveContainer" containerID="1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"
Oct 31 00:22:32.629290 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.629252    6033 scope.go:122] "RemoveContainer" containerID="1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"
Oct 31 00:22:32.630284 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:32.630199    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3\": container with ID starting with 1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3 not found: ID does not exist" containerID="1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"
Oct 31 00:22:32.630284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.630245    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3"} err="failed to get container status \"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3\": rpc error: code = NotFound desc = could not find container \"1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3\": container with ID starting with 1474b11038f72587e4a95c578aee13fe7893832fc72df5d4354a0bfbfd8824c3 not found: ID does not exist"
Oct 31 00:22:32.773518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.773474    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 31 00:22:32.777733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:32.777698    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-35a187-59b76ccb5b-25xkv"]
Oct 31 00:22:33.831195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:33.831162    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7387d560-e431-4455-82ea-20cae348ed33" path="/var/lib/kubelet/pods/7387d560-e431-4455-82ea-20cae348ed33/volumes"
Oct 31 00:22:36.658832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.653863    6033 generic.go:356] "Generic (PLEG): container finished" podID="476bf414-e7cc-48f3-b062-6827959edc9f" containerID="150055cfa5a6a4b9d7e4866b480002a54f54f8d2830c3cf790392d2e8c952881" exitCode=0
Oct 31 00:22:36.658832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.654009    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" event={"ID":"476bf414-e7cc-48f3-b062-6827959edc9f","Type":"ContainerDied","Data":"150055cfa5a6a4b9d7e4866b480002a54f54f8d2830c3cf790392d2e8c952881"}
Oct 31 00:22:36.745144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.745112    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 31 00:22:36.845746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.845708    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.845967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.845821    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.845967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.845872    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.845967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.845905    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.845967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.845955    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.846258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846012    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.846258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846121    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.846258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846159    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.846258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846199    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.846258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846219    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\") pod \"476bf414-e7cc-48f3-b062-6827959edc9f\" (UID: \"476bf414-e7cc-48f3-b062-6827959edc9f\") "
Oct 31 00:22:36.846501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846368    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.846501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846387    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.846501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846426    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.846777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846756    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:36.846916 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846898    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.846978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846920    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.846978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846935    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.846978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846948    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.846978 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.846963    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/476bf414-e7cc-48f3-b062-6827959edc9f-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.848374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.848344    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:36.848479 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.848428    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "kube-api-access-bfcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:36.848582 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.848556    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:22:36.848681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.848603    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token" pod "476bf414-e7cc-48f3-b062-6827959edc9f" (UID: "476bf414-e7cc-48f3-b062-6827959edc9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:36.948888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.948796    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.948888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.948828    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcz5\" (UniqueName: \"kubernetes.io/projected/476bf414-e7cc-48f3-b062-6827959edc9f-kube-api-access-bfcz5\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.948888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.948843    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/476bf414-e7cc-48f3-b062-6827959edc9f-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:36.948888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:36.948855    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/476bf414-e7cc-48f3-b062-6827959edc9f-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:37.662058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:37.662006    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"
Oct 31 00:22:37.662525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:37.662009    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8" event={"ID":"476bf414-e7cc-48f3-b062-6827959edc9f","Type":"ContainerDied","Data":"982ca16c723ff7763839279f7a8cd2551478816aeaaab7aa0b404f769f7aae00"}
Oct 31 00:22:37.662525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:37.662201    6033 scope.go:122] "RemoveContainer" containerID="150055cfa5a6a4b9d7e4866b480002a54f54f8d2830c3cf790392d2e8c952881"
Oct 31 00:22:37.819735 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:37.819695    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 31 00:22:37.866299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:37.866253    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fe01fa-openshift-default-564f5c6485-8hzf8"]
Oct 31 00:22:38.755378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:38.755337    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:22:38.756024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:38.755604    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" containerID="cri-o://5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f" gracePeriod=30
Oct 31 00:22:39.572621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:39.572568    6033 patch_prober.go:29] interesting pod/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:22:39.572799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:39.572649    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:22:39.829821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:39.829737    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" path="/var/lib/kubelet/pods/476bf414-e7cc-48f3-b062-6827959edc9f/volumes"
Oct 31 00:22:40.223318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:40.223233    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 31 00:22:40.684150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:40.684108    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy" containerID="cri-o://139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3" gracePeriod=30
Oct 31 00:22:44.042910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.042879    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:22:44.154763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.154707    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.154968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.154810    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.154968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.154906    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.154968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.154936    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.154987    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155020    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:44.155174 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155066    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155176    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155252    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155312    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\") pod \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\" (UID: \"6966b8f0-0cbc-45d7-ae91-73f210fc3a39\") "
Oct 31 00:22:44.155586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155553    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:44.155732 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155705    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:44.155800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155771    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:44.155903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155883    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:44.156006 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.155987    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.156092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.156009    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.156092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.156025    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.156092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.156065    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.156092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.156078    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.157450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.157425    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:44.157450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.157439    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:22:44.157652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.157636    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "kube-api-access-ll6rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:44.157692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.157666    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy" pod "6966b8f0-0cbc-45d7-ae91-73f210fc3a39" (UID: "6966b8f0-0cbc-45d7-ae91-73f210fc3a39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:44.257009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.256969    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6rg\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-kube-api-access-ll6rg\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.257009 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.257010    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.257257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.257063    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.257257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.257077    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6966b8f0-0cbc-45d7-ae91-73f210fc3a39-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:44.713327 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.713294    6033 generic.go:356] "Generic (PLEG): container finished" podID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerID="5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f" exitCode=0
Oct 31 00:22:44.713327 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.713341    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"
Oct 31 00:22:44.713608 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.713380    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" event={"ID":"6966b8f0-0cbc-45d7-ae91-73f210fc3a39","Type":"ContainerDied","Data":"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"}
Oct 31 00:22:44.713608 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.713440    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs" event={"ID":"6966b8f0-0cbc-45d7-ae91-73f210fc3a39","Type":"ContainerDied","Data":"d4bf2fc915948deab33fb83b943adb042bc24828a63329b572e67362e84788c9"}
Oct 31 00:22:44.713608 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.713470    6033 scope.go:122] "RemoveContainer" containerID="5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"
Oct 31 00:22:44.725057 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.725023    6033 scope.go:122] "RemoveContainer" containerID="5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"
Oct 31 00:22:44.725437 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:44.725415    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f\": container with ID starting with 5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f not found: ID does not exist" containerID="5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"
Oct 31 00:22:44.725483 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.725447    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f"} err="failed to get container status \"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f\": rpc error: code = NotFound desc = could not find container \"5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f\": container with ID starting with 5ef46116d825c567af1122fb8b59e638964056cf21b8698d16cc01ee46ab3d4f not found: ID does not exist"
Oct 31 00:22:44.927092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.927056    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:22:44.973885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:44.973789    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-986d37-openshift-default-5bc8bd86fc-t28zs"]
Oct 31 00:22:45.831986 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.831950    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" path="/var/lib/kubelet/pods/6966b8f0-0cbc-45d7-ae91-73f210fc3a39/volumes"
Oct 31 00:22:45.954537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.954454    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 31 00:22:45.979181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979141    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979196    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979249    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979276    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979321    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979390 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979382    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979443    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979475    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979511    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:45.979683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979600    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\") pod \"05295f90-3b8c-4406-b5ee-dab044069376\" (UID: \"05295f90-3b8c-4406-b5ee-dab044069376\") "
Oct 31 00:22:45.979683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979642    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:45.979970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979875    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:45.979970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.979944    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:45.980107 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980004    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:22:45.980225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980201    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:45.980225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980227    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:45.980417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980241    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:45.980417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980255    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:45.980417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.980268    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05295f90-3b8c-4406-b5ee-dab044069376-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:45.981877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.981838    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "kube-api-access-gkqmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:45.981877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.981869    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:22:45.982104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.982003    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:22:45.982104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:45.982062    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token" pod "05295f90-3b8c-4406-b5ee-dab044069376" (UID: "05295f90-3b8c-4406-b5ee-dab044069376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:22:46.081149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.081106    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05295f90-3b8c-4406-b5ee-dab044069376-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:46.081149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.081143    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:46.081149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.081156    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05295f90-3b8c-4406-b5ee-dab044069376-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:46.081411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.081171    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqmw\" (UniqueName: \"kubernetes.io/projected/05295f90-3b8c-4406-b5ee-dab044069376-kube-api-access-gkqmw\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:22:46.730010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.729969    6033 generic.go:356] "Generic (PLEG): container finished" podID="05295f90-3b8c-4406-b5ee-dab044069376" containerID="139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3" exitCode=0
Oct 31 00:22:46.730204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.730056    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" event={"ID":"05295f90-3b8c-4406-b5ee-dab044069376","Type":"ContainerDied","Data":"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"}
Oct 31 00:22:46.730204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.730079    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"
Oct 31 00:22:46.730204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.730166    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6" event={"ID":"05295f90-3b8c-4406-b5ee-dab044069376","Type":"ContainerDied","Data":"3196c1fd4071f786ed4bfb97d4c08322bbc8acbd38334e99514317848b8a057d"}
Oct 31 00:22:46.730204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.730195    6033 scope.go:122] "RemoveContainer" containerID="139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"
Oct 31 00:22:46.741616 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.741587    6033 scope.go:122] "RemoveContainer" containerID="139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"
Oct 31 00:22:46.741952 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:46.741930    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3\": container with ID starting with 139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3 not found: ID does not exist" containerID="139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"
Oct 31 00:22:46.742054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.741962    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3"} err="failed to get container status \"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3\": rpc error: code = NotFound desc = could not find container \"139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3\": container with ID starting with 139dfd58b004aaea9e1c3f248264c6ee58c4fc4ad87c64d1a7286b5feb74bfc3 not found: ID does not exist"
Oct 31 00:22:46.887042 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.886989    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 31 00:22:46.933543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:46.933504    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-6hww6"]
Oct 31 00:22:46.970779 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:22:46.970746    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:22:47.830113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:47.830078    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05295f90-3b8c-4406-b5ee-dab044069376" path="/var/lib/kubelet/pods/05295f90-3b8c-4406-b5ee-dab044069376/volumes"
Oct 31 00:22:50.771884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:22:50.771846    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:23:08.624458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.624370    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"]
Oct 31 00:23:08.625062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625016    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5451011c-f57a-4c3e-9d82-455788d97bec" containerName="revoke-keys"
Oct 31 00:23:08.625062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625047    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5451011c-f57a-4c3e-9d82-455788d97bec" containerName="revoke-keys"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625075    6033 cpu_manager.go:414] "containerMap: removing container" podUID="49b150ef-b32d-4dcc-abb1-da37a970d023" containerName="revoke-keys"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625081    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49b150ef-b32d-4dcc-abb1-da37a970d023" containerName="revoke-keys"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625095    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625101    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625108    6033 cpu_manager.go:414] "containerMap: removing container" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625113    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625123    6033 cpu_manager.go:414] "containerMap: removing container" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625127    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625140    6033 cpu_manager.go:414] "containerMap: removing container" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main"
Oct 31 00:23:08.625149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625145    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625158    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625163    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625248    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5451011c-f57a-4c3e-9d82-455788d97bec" containerName="revoke-keys"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625260    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="49b150ef-b32d-4dcc-abb1-da37a970d023" containerName="revoke-keys"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625267    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7387d560-e431-4455-82ea-20cae348ed33" containerName="payload-processing"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625273    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="05295f90-3b8c-4406-b5ee-dab044069376" containerName="istio-proxy"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625282    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="476bf414-e7cc-48f3-b062-6827959edc9f" containerName="istio-proxy"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625290    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d8639e-c06b-49e2-82f0-166c0308f907" containerName="main"
Oct 31 00:23:08.625445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.625298    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6966b8f0-0cbc-45d7-ae91-73f210fc3a39" containerName="istio-proxy"
Oct 31 00:23:08.629297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.629271    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.679766 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.679723    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:23:08.693486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.693447    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzgb\" (UniqueName: \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.693693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.693666    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.693757 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.693706    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.726402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.726356    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:23:08.729155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.729129    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"]
Oct 31 00:23:08.795373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.795333    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.795559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.795386    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.795613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.795562    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzgb\" (UniqueName: \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.796263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.796237    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.797888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.797866    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.853723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.853687    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzgb\" (UniqueName: \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\") pod \"maas-api-revoke-keys-e2e-api-a-92e375-w4rbb\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:08.943195 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:08.943100    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:09.316876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:09.316848    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"]
Oct 31 00:23:09.319366 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:23:09.319337    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a45fa67_9b81_4493_96d5_91ea2aa61947.slice/crio-e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43 WatchSource:0}: Error finding container e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43: Status 404 returned error can't find the container with id e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43
Oct 31 00:23:09.907335 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:09.907293    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb" event={"ID":"7a45fa67-9b81-4493-96d5-91ea2aa61947","Type":"ContainerStarted","Data":"68b83541606662a8f01cd942ef5d9b734bd4a027716fb85617fb8eea1ee4949c"}
Oct 31 00:23:09.907824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:09.907354    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb" event={"ID":"7a45fa67-9b81-4493-96d5-91ea2aa61947","Type":"ContainerStarted","Data":"e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43"}
Oct 31 00:23:10.017213 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:10.017155    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb" podStartSLOduration=2.017138699 podStartE2EDuration="2.017138699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:10.016777533 +0000 UTC m=+2442.363761431" watchObservedRunningTime="2026-10-31 00:23:10.017138699 +0000 UTC m=+2442.364122597"
Oct 31 00:23:10.916061 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:10.915996    6033 generic.go:356] "Generic (PLEG): container finished" podID="7a45fa67-9b81-4493-96d5-91ea2aa61947" containerID="68b83541606662a8f01cd942ef5d9b734bd4a027716fb85617fb8eea1ee4949c" exitCode=0
Oct 31 00:23:10.916489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:10.916061    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb" event={"ID":"7a45fa67-9b81-4493-96d5-91ea2aa61947","Type":"ContainerDied","Data":"68b83541606662a8f01cd942ef5d9b734bd4a027716fb85617fb8eea1ee4949c"}
Oct 31 00:23:12.089564 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.089536    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:12.237787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.237628    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\" (UniqueName: \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\") pod \"7a45fa67-9b81-4493-96d5-91ea2aa61947\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") "
Oct 31 00:23:12.237989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.237786    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\") pod \"7a45fa67-9b81-4493-96d5-91ea2aa61947\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") "
Oct 31 00:23:12.238091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.237992    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\") pod \"7a45fa67-9b81-4493-96d5-91ea2aa61947\" (UID: \"7a45fa67-9b81-4493-96d5-91ea2aa61947\") "
Oct 31 00:23:12.238259 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.238220    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca" pod "7a45fa67-9b81-4493-96d5-91ea2aa61947" (UID: "7a45fa67-9b81-4493-96d5-91ea2aa61947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:12.238585 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.238559    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:12.239912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.239882    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb" pod "7a45fa67-9b81-4493-96d5-91ea2aa61947" (UID: "7a45fa67-9b81-4493-96d5-91ea2aa61947"). InnerVolumeSpecName "kube-api-access-7xzgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:12.240406 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.240385    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca" pod "7a45fa67-9b81-4493-96d5-91ea2aa61947" (UID: "7a45fa67-9b81-4493-96d5-91ea2aa61947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:23:12.340329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.340282    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a45fa67-9b81-4493-96d5-91ea2aa61947-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:12.340329 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.340329    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzgb\" (UniqueName: \"kubernetes.io/projected/7a45fa67-9b81-4493-96d5-91ea2aa61947-kube-api-access-7xzgb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:12.935963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.935924    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb" event={"ID":"7a45fa67-9b81-4493-96d5-91ea2aa61947","Type":"ContainerDied","Data":"e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43"}
Oct 31 00:23:12.935963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.935956    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"
Oct 31 00:23:12.936257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:12.935976    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37dbdda71d76f0165864c0ce6d11d88cb498b18131bb352a215a668ab8dfb43"
Oct 31 00:23:15.070514 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:15.070477    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"]
Oct 31 00:23:15.117297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:15.117259    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-257e4c78-wvtxl"]
Oct 31 00:23:15.831089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:15.831052    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6" path="/var/lib/kubelet/pods/2f0588f8-9a1e-44f3-af1c-b5f3aa5895d6/volumes"
Oct 31 00:23:16.971358 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:23:16.971325    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:23:21.532148 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.532112    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:23:21.532816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.532799    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7a45fa67-9b81-4493-96d5-91ea2aa61947" containerName="revoke-keys"
Oct 31 00:23:21.532868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.532817    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a45fa67-9b81-4493-96d5-91ea2aa61947" containerName="revoke-keys"
Oct 31 00:23:21.532931 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.532919    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a45fa67-9b81-4493-96d5-91ea2aa61947" containerName="revoke-keys"
Oct 31 00:23:21.536707 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.536685    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.589236 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.589190    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-37abc0-serving-cert\""
Oct 31 00:23:21.638640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.638600    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:23:21.670300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.664278    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.670300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.664527    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.670300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.664688    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5x4n\" (UniqueName: \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.670300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.664742    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.766336 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.766294    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5x4n\" (UniqueName: \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.766569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.766355    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.766569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.766434    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.766569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.766564    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.767293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.767265    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.769016 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.768984    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.769154 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.769055    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.826860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.826783    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5x4n\" (UniqueName: \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\") pod \"maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:21.851720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:21.851684    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:22.228592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:22.228557    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:23:22.230114 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:23:22.230086    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc9fd65_4b35_4a84_9c04_64d03ae28fe5.slice/crio-11a8f5eba1d666dde554c4a474ab0ecb25d770779977ade7d5ee54a4c50afa18 WatchSource:0}: Error finding container 11a8f5eba1d666dde554c4a474ab0ecb25d770779977ade7d5ee54a4c50afa18: Status 404 returned error can't find the container with id 11a8f5eba1d666dde554c4a474ab0ecb25d770779977ade7d5ee54a4c50afa18
Oct 31 00:23:23.009651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:23.009612    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" event={"ID":"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5","Type":"ContainerStarted","Data":"11a8f5eba1d666dde554c4a474ab0ecb25d770779977ade7d5ee54a4c50afa18"}
Oct 31 00:23:24.018665 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:24.018625    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" event={"ID":"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5","Type":"ContainerStarted","Data":"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"}
Oct 31 00:23:24.019123 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:24.018834    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:24.131187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:24.131130    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" podStartSLOduration=2.183413323 podStartE2EDuration="3.131114071s" totalImagesPullingTime="947.700748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:24.129624584 +0000 UTC m=+2456.476608473" watchObservedRunningTime="2026-10-31 00:23:24.131114071 +0000 UTC m=+2456.478097969"
Oct 31 00:23:27.904973 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:27.904933    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"]
Oct 31 00:23:27.951185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:27.951140    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-35a187-gc7pg"]
Oct 31 00:23:29.415968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.415932    6033 scope.go:122] "RemoveContainer" containerID="aa60b25cf498d8a4573f1b132d6c61da9957afef543fe54d7aff699977c0d56f"
Oct 31 00:23:29.426676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.426650    6033 scope.go:122] "RemoveContainer" containerID="3295d2284918c6566e06335254d8ff1aed593e29f880034ff1a8c21ed3f5696f"
Oct 31 00:23:29.437182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.437157    6033 scope.go:122] "RemoveContainer" containerID="a2d928b462319f9582e79a7c5b97f1a12cc4294271207def8e339475077479aa"
Oct 31 00:23:29.448861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.448835    6033 scope.go:122] "RemoveContainer" containerID="46d4ae82b16458aad78458adf93942ae80013651c8d225ee57a913f0cbfd6f98"
Oct 31 00:23:29.461378 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.461247    6033 scope.go:122] "RemoveContainer" containerID="c798f9ebcbdc1af93fae35036ecedf56b264e1648acf5eafdf0627808b464c83"
Oct 31 00:23:29.473417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.473388    6033 scope.go:122] "RemoveContainer" containerID="28a4b5e978d211a7be64e4fba18802cdeedf11539a716fb37ef823026b6351d3"
Oct 31 00:23:29.483880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.483854    6033 scope.go:122] "RemoveContainer" containerID="68168032c07dc9cd667c2bfd45638c93ea4d37b4cec56ec0754ec0e9c6bb8506"
Oct 31 00:23:29.830528 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:29.830492    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae69c5b-a6b4-4015-aa32-d7e2270f5e5e" path="/var/lib/kubelet/pods/fae69c5b-a6b4-4015-aa32-d7e2270f5e5e/volumes"
Oct 31 00:23:30.031627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:30.031593    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:23:32.623216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.623177    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:23:32.629742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.629708    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705276 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705231    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705575 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705378    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705575 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705473    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705618    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705648    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzn4\" (UniqueName: \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.705750 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.705685    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.726586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.726552    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76b073\"/\"openshift-service-ca.crt\""
Oct 31 00:23:32.726586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.726550    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76b073\"/\"kube-root-ca.crt\""
Oct 31 00:23:32.726848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.726688    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76b073\"/\"test-model-76b073-kserve-self-signed-certs\""
Oct 31 00:23:32.726848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.726768    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76b073\"/\"default-dockercfg-mncq2\""
Oct 31 00:23:32.729714 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.729679    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:23:32.806814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806776    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.806814 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806824    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzn4\" (UniqueName: \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806855    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806909    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806970    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.806996    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807426 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.807405    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.807408    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.807511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.807435    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.809348 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.809317    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.809648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.809627    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.865852 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.865809    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzn4\" (UniqueName: \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\") pod \"test-model-76b073-kserve-74b8cc9c7c-rp6qs\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") " pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:32.943136 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:32.943015    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:33.357653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:33.357622    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:23:33.359344 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:23:33.359314    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf562d09b_d562_4c4a_8aaf_dcd67f0ce452.slice/crio-9e5bb8aa2e439c033372e2dd668e343447026840394c6d6eec2fe501322bbb72 WatchSource:0}: Error finding container 9e5bb8aa2e439c033372e2dd668e343447026840394c6d6eec2fe501322bbb72: Status 404 returned error can't find the container with id 9e5bb8aa2e439c033372e2dd668e343447026840394c6d6eec2fe501322bbb72
Oct 31 00:23:34.103498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:34.103452    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" event={"ID":"f562d09b-d562-4c4a-8aaf-dcd67f0ce452","Type":"ContainerStarted","Data":"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"}
Oct 31 00:23:34.103926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:34.103514    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" event={"ID":"f562d09b-d562-4c4a-8aaf-dcd67f0ce452","Type":"ContainerStarted","Data":"9e5bb8aa2e439c033372e2dd668e343447026840394c6d6eec2fe501322bbb72"}
Oct 31 00:23:34.943628 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:34.943586    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:35.363983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:35.363922    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" podStartSLOduration=3.363904891 podStartE2EDuration="3.363904891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:34.258456133 +0000 UTC m=+2466.605440031" watchObservedRunningTime="2026-10-31 00:23:35.363904891 +0000 UTC m=+2467.710888788"
Oct 31 00:23:35.365903 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:35.365874    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 31 00:23:35.366176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:35.366144    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy" containerID="cri-o://f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3" gracePeriod=30
Oct 31 00:23:40.660795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.660757    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 31 00:23:40.840289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840255    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840323    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840416    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840451    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840504 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840488    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840599    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840606    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.840728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840685    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840741    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.840890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840782    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\") pod \"3f4cecc1-50da-4e73-84e6-656bf87266e1\" (UID: \"3f4cecc1-50da-4e73-84e6-656bf87266e1\") "
Oct 31 00:23:40.841334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.840780    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.841334 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.841318    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.841487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.841328    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.841487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.841405    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.841594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.841568    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:23:40.842936 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.842905    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:40.843258 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.843229    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:23:40.843402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.843382    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "kube-api-access-4zs7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:23:40.843858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.843836    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy" pod "3f4cecc1-50da-4e73-84e6-656bf87266e1" (UID: "3f4cecc1-50da-4e73-84e6-656bf87266e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:23:40.943800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943699    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zs7p\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-kube-api-access-4zs7p\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.943800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943752    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.943800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943767    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.943800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943783    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.943800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943798    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.944194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943817    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4cecc1-50da-4e73-84e6-656bf87266e1-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.944194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943831    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4cecc1-50da-4e73-84e6-656bf87266e1-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:40.944194 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:40.943847    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cecc1-50da-4e73-84e6-656bf87266e1-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:23:41.159062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.158991    6033 generic.go:356] "Generic (PLEG): container finished" podID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerID="f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3" exitCode=0
Oct 31 00:23:41.159252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.159068    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"
Oct 31 00:23:41.159252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.159090    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" event={"ID":"3f4cecc1-50da-4e73-84e6-656bf87266e1","Type":"ContainerDied","Data":"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"}
Oct 31 00:23:41.159252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.159180    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk" event={"ID":"3f4cecc1-50da-4e73-84e6-656bf87266e1","Type":"ContainerDied","Data":"c84dbca76c2ae229cc5f14cea8bc62d0b986ed0dfee1275dfc6f66dd24ea80f9"}
Oct 31 00:23:41.159252 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.159204    6033 scope.go:122] "RemoveContainer" containerID="f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"
Oct 31 00:23:41.170776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.170754    6033 scope.go:122] "RemoveContainer" containerID="f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"
Oct 31 00:23:41.171119 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:23:41.171099    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3\": container with ID starting with f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3 not found: ID does not exist" containerID="f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"
Oct 31 00:23:41.171179 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.171128    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3"} err="failed to get container status \"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3\": rpc error: code = NotFound desc = could not find container \"f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3\": container with ID starting with f78132d367a43b03ade074d0896d02d2d550b3de68e85bfacefb74c5d6231ff3 not found: ID does not exist"
Oct 31 00:23:41.317096 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.317062    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 31 00:23:41.363185 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.363146    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-35a187-openshift-default-5b54c65f75-bsqhk"]
Oct 31 00:23:41.830805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:41.830768    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" path="/var/lib/kubelet/pods/3f4cecc1-50da-4e73-84e6-656bf87266e1/volumes"
Oct 31 00:23:42.943499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:42.943450    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:42.956556 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:42.956523    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:43.189645 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:43.189608    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:23:43.333807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:43.333763    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:23:46.978495 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:23:46.978454    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:23:54.483325 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.483287    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:23:54.483918 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.483903    6033 cpu_manager.go:414] "containerMap: removing container" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy"
Oct 31 00:23:54.483963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.483919    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy"
Oct 31 00:23:54.484064 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.484054    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4cecc1-50da-4e73-84e6-656bf87266e1" containerName="istio-proxy"
Oct 31 00:23:54.487948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.487921    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.586017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.585977    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49837c\"/\"openshift-service-ca.crt\""
Oct 31 00:23:54.586289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.586076    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49837c\"/\"kube-root-ca.crt\""
Oct 31 00:23:54.586289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.586168    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49837c\"/\"default-dockercfg-ljgbw\""
Oct 31 00:23:54.586289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.586172    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49837c\"/\"test-model-49837c-kserve-self-signed-certs\""
Oct 31 00:23:54.589120 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.589085    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:23:54.620003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.619954    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.620003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.620004    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.620279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.620078    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.620279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.620195    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wv6z\" (UniqueName: \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.620279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.620233    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.620279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.620249    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721416    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721548    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wv6z\" (UniqueName: \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721607    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721631    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721746    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.721886 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.721787    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.722129 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.722101    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.722211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.722129    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.722273 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.722211    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.724092 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.724066    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.724248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.724237    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.779647 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.779604    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wv6z\" (UniqueName: \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\") pod \"test-model-49837c-kserve-59b78788cf-kvqx6\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") " pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:54.801488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:54.801444    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:23:55.214023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:55.213993    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:23:55.216116 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:23:55.216072    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ebe9a06_4625_43ea_b41b_15f419f8d292.slice/crio-050e2d2d6fc68dd279d8805ea6ad52e55392b1a4a07f1151173a9ee451d16480 WatchSource:0}: Error finding container 050e2d2d6fc68dd279d8805ea6ad52e55392b1a4a07f1151173a9ee451d16480: Status 404 returned error can't find the container with id 050e2d2d6fc68dd279d8805ea6ad52e55392b1a4a07f1151173a9ee451d16480
Oct 31 00:23:55.264849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:55.264808    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" event={"ID":"2ebe9a06-4625-43ea-b41b-15f419f8d292","Type":"ContainerStarted","Data":"050e2d2d6fc68dd279d8805ea6ad52e55392b1a4a07f1151173a9ee451d16480"}
Oct 31 00:23:56.273700 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:56.273661    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" event={"ID":"2ebe9a06-4625-43ea-b41b-15f419f8d292","Type":"ContainerStarted","Data":"29533d084c12de91b6fdf863a5edc32dad36ba17f67948564621a17151977e28"}
Oct 31 00:23:56.386508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:56.386445    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" podStartSLOduration=2.386429519 podStartE2EDuration="2.386429519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:56.384560684 +0000 UTC m=+2488.731544582" watchObservedRunningTime="2026-10-31 00:23:56.386429519 +0000 UTC m=+2488.733413427"
Oct 31 00:23:56.802156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:23:56.802117    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:24:04.802244 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:04.802194    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:24:04.815192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:04.815155    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:24:05.354783 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:05.354743    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:24:16.970916 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:24:16.970876    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:24:26.940806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:26.940763    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:24:29.568782 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:29.568739    6033 scope.go:122] "RemoveContainer" containerID="cb4a21f07a67220287874d7d681f283f293d70054fce62c41a823baedcfd4aa2"
Oct 31 00:24:41.755821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.755730    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:24:41.762153 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.762124    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.822687 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.822646    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t677v\" (UniqueName: \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.822892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.822769    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.822892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.822832    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.822979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.822927    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.822979 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.822967    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.823179 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.823159    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.835370 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.835330    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:24:41.858200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.858153    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddf61a\"/\"default-dockercfg-9tvjn\""
Oct 31 00:24:41.858200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.858153    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddf61a\"/\"kube-root-ca.crt\""
Oct 31 00:24:41.858200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.858169    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ddf61a\"/\"openshift-service-ca.crt\""
Oct 31 00:24:41.858200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.858201    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ddf61a\"/\"test-model-ddf61a-kserve-self-signed-certs\""
Oct 31 00:24:41.924204 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924161    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924305    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924329    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924488    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924597    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t677v\" (UniqueName: \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924680    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924735    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.924815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.924798    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.925079 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.925054    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.926640 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.926614    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.926962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.926942    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:41.982610 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:41.982571    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t677v\" (UniqueName: \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\") pod \"test-model-ddf61a-kserve-5bb9f849c5-9sjqg\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") " pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:42.075666 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:42.075554    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:42.460060 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:42.460014    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:24:42.462118 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:24:42.462083    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfac9a388_5d54_453f_bf9d_71aabb9bb90e.slice/crio-25d86b6c1d670e17799d51838d65bc47dd91cd729b8039910fa50e312de047ff WatchSource:0}: Error finding container 25d86b6c1d670e17799d51838d65bc47dd91cd729b8039910fa50e312de047ff: Status 404 returned error can't find the container with id 25d86b6c1d670e17799d51838d65bc47dd91cd729b8039910fa50e312de047ff
Oct 31 00:24:42.615949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:42.615906    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" event={"ID":"fac9a388-5d54-453f-bf9d-71aabb9bb90e","Type":"ContainerStarted","Data":"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"}
Oct 31 00:24:42.616163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:42.615960    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" event={"ID":"fac9a388-5d54-453f-bf9d-71aabb9bb90e","Type":"ContainerStarted","Data":"25d86b6c1d670e17799d51838d65bc47dd91cd729b8039910fa50e312de047ff"}
Oct 31 00:24:42.729143 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:42.729012    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" podStartSLOduration=1.72899744 podStartE2EDuration="1.72899744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:42.727229152 +0000 UTC m=+2535.074213052" watchObservedRunningTime="2026-10-31 00:24:42.72899744 +0000 UTC m=+2535.075981335"
Oct 31 00:24:43.076547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:43.076503    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:46.970387 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:24:46.970340    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:24:52.075948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:52.075906    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:52.092568 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:52.092534    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:52.703756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:52.703722    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:24:58.477581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:58.477533    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"]
Oct 31 00:24:58.524364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:58.524324    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-92e375-w4rbb"]
Oct 31 00:24:59.721082 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:59.721011    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:24:59.838544 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:24:59.838507    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a45fa67-9b81-4493-96d5-91ea2aa61947" path="/var/lib/kubelet/pods/7a45fa67-9b81-4493-96d5-91ea2aa61947/volumes"
Oct 31 00:25:16.972646 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:25:16.972605    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:25:17.934175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:17.934129    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:25:18.033854 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:18.033814    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:25:22.935646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:22.935561    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" podUID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" containerName="payload-processing" containerID="cri-o://fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802" gracePeriod=25
Oct 31 00:25:23.034333 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.034267    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" podUID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" containerName="payload-pre-processing" containerID="cri-o://a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3" gracePeriod=25
Oct 31 00:25:23.238917 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.238882    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:25:23.267000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.266966    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\" (UniqueName: \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\") pod \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") "
Oct 31 00:25:23.267452 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.267431    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\") pod \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\" (UID: \"ad36225a-ea11-4d9a-8efd-8e917933bc5f\") "
Oct 31 00:25:23.267869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.267836    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume" pod "ad36225a-ea11-4d9a-8efd-8e917933bc5f" (UID: "ad36225a-ea11-4d9a-8efd-8e917933bc5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:25:23.268000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.267980    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad36225a-ea11-4d9a-8efd-8e917933bc5f-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:23.270322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.270289    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv" pod "ad36225a-ea11-4d9a-8efd-8e917933bc5f" (UID: "ad36225a-ea11-4d9a-8efd-8e917933bc5f"). InnerVolumeSpecName "kube-api-access-l7hfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:25:23.311114 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.311082    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:25:23.368839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.368802    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\") pod \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") "
Oct 31 00:25:23.369224 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.368919    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\" (UniqueName: \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\") pod \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\" (UID: \"0a9783b1-d7b7-44a1-8bef-8a323435aee2\") "
Oct 31 00:25:23.369384 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.369270    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7hfv\" (UniqueName: \"kubernetes.io/projected/ad36225a-ea11-4d9a-8efd-8e917933bc5f-kube-api-access-l7hfv\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:23.369384 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.369350    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume" pod "0a9783b1-d7b7-44a1-8bef-8a323435aee2" (UID: "0a9783b1-d7b7-44a1-8bef-8a323435aee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:25:23.371313 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.371257    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh" pod "0a9783b1-d7b7-44a1-8bef-8a323435aee2" (UID: "0a9783b1-d7b7-44a1-8bef-8a323435aee2"). InnerVolumeSpecName "kube-api-access-rgfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:25:23.470678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.470591    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a9783b1-d7b7-44a1-8bef-8a323435aee2-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:23.470678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.470623    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfjh\" (UniqueName: \"kubernetes.io/projected/0a9783b1-d7b7-44a1-8bef-8a323435aee2-kube-api-access-rgfjh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:23.915098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.915065    6033 generic.go:356] "Generic (PLEG): container finished" podID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" containerID="fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802" exitCode=0
Oct 31 00:25:23.915098 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.915102    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"
Oct 31 00:25:23.915365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.915137    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" event={"ID":"ad36225a-ea11-4d9a-8efd-8e917933bc5f","Type":"ContainerDied","Data":"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"}
Oct 31 00:25:23.915365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.915194    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj" event={"ID":"ad36225a-ea11-4d9a-8efd-8e917933bc5f","Type":"ContainerDied","Data":"21c5671e6ab3d6eac5ff82e9f168920f628b6d7512f806a2278cb5530966c474"}
Oct 31 00:25:23.915365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.915224    6033 scope.go:122] "RemoveContainer" containerID="fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"
Oct 31 00:25:23.916893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.916866    6033 generic.go:356] "Generic (PLEG): container finished" podID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" containerID="a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3" exitCode=0
Oct 31 00:25:23.917023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.916922    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" event={"ID":"0a9783b1-d7b7-44a1-8bef-8a323435aee2","Type":"ContainerDied","Data":"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"}
Oct 31 00:25:23.917023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.916929    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"
Oct 31 00:25:23.917023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.916959    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz" event={"ID":"0a9783b1-d7b7-44a1-8bef-8a323435aee2","Type":"ContainerDied","Data":"a42fb2daa2bb54376b4c769cd280e48b41597ea35a809463da9f3efbdcbfcc3e"}
Oct 31 00:25:23.928070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.928011    6033 scope.go:122] "RemoveContainer" containerID="fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"
Oct 31 00:25:23.929600 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:25:23.929568    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802\": container with ID starting with fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802 not found: ID does not exist" containerID="fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"
Oct 31 00:25:23.929752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.929607    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802"} err="failed to get container status \"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802\": rpc error: code = NotFound desc = could not find container \"fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802\": container with ID starting with fc21c1d44230691cceee453533219395d0467d9fb3c4e47c1dc2df78d97df802 not found: ID does not exist"
Oct 31 00:25:23.929752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.929637    6033 scope.go:122] "RemoveContainer" containerID="a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"
Oct 31 00:25:23.940509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.940330    6033 scope.go:122] "RemoveContainer" containerID="a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"
Oct 31 00:25:23.940836 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:25:23.940724    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3\": container with ID starting with a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3 not found: ID does not exist" containerID="a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"
Oct 31 00:25:23.940836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:23.940751    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3"} err="failed to get container status \"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3\": rpc error: code = NotFound desc = could not find container \"a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3\": container with ID starting with a327c390e4e40051a5723f9f3de99a0b29763780e0ae8c15cb21971fa625e0a3 not found: ID does not exist"
Oct 31 00:25:24.071489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:24.071444    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:25:24.118089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:24.117972    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d687ff-7666858cfd-wh6tj"]
Oct 31 00:25:24.181278 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:24.181240    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:25:24.186003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:24.185968    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d687ff-7c6c95ffc9-4xkwz"]
Oct 31 00:25:25.830318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:25.830281    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" path="/var/lib/kubelet/pods/0a9783b1-d7b7-44a1-8bef-8a323435aee2/volumes"
Oct 31 00:25:25.830745 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:25.830730    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" path="/var/lib/kubelet/pods/ad36225a-ea11-4d9a-8efd-8e917933bc5f/volumes"
Oct 31 00:25:29.602587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:29.602539    6033 scope.go:122] "RemoveContainer" containerID="7f510d8d175dc8b1438a8c06264d58e99355a3dabcea61eede1983e3db2eb39b"
Oct 31 00:25:33.637716 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:33.637672    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:25:40.046637 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:40.046576    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"]
Oct 31 00:25:40.094567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:40.094524    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff797a69-k4fz9"]
Oct 31 00:25:41.830135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:41.830101    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44320c59-e6db-4362-ad41-83eafcaca1bb" path="/var/lib/kubelet/pods/44320c59-e6db-4362-ad41-83eafcaca1bb/volumes"
Oct 31 00:25:44.048636 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.048590    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:25:44.049165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.048866    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerName="maas-api" containerID="cri-o://809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad" gracePeriod=30
Oct 31 00:25:44.377976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.377940    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:25:44.446569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.446523    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\") pod \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") "
Oct 31 00:25:44.446765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.446602    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\") pod \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") "
Oct 31 00:25:44.446858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.446781    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\" (UniqueName: \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\") pod \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") "
Oct 31 00:25:44.447930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.446946    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") pod \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\" (UID: \"b254d093-f40a-4d3d-ab73-7e3c4694a5cd\") "
Oct 31 00:25:44.448656 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.448494    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca" pod "b254d093-f40a-4d3d-ab73-7e3c4694a5cd" (UID: "b254d093-f40a-4d3d-ab73-7e3c4694a5cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:25:44.452224 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.452179    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8" pod "b254d093-f40a-4d3d-ab73-7e3c4694a5cd" (UID: "b254d093-f40a-4d3d-ab73-7e3c4694a5cd"). InnerVolumeSpecName "kube-api-access-dzgq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:25:44.453909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.453870    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs" pod "b254d093-f40a-4d3d-ab73-7e3c4694a5cd" (UID: "b254d093-f40a-4d3d-ab73-7e3c4694a5cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:25:44.453909 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.453894    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls" pod "b254d093-f40a-4d3d-ab73-7e3c4694a5cd" (UID: "b254d093-f40a-4d3d-ab73-7e3c4694a5cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:25:44.549229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.549191    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:44.549229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.549225    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:44.549229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.549236    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:44.549493 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:44.549245    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgq8\" (UniqueName: \"kubernetes.io/projected/b254d093-f40a-4d3d-ab73-7e3c4694a5cd-kube-api-access-dzgq8\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:25:45.063852 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.063816    6033 generic.go:356] "Generic (PLEG): container finished" podID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerID="809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad" exitCode=0
Oct 31 00:25:45.064326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.063866    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"
Oct 31 00:25:45.064326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.063901    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" event={"ID":"b254d093-f40a-4d3d-ab73-7e3c4694a5cd","Type":"ContainerDied","Data":"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"}
Oct 31 00:25:45.064326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.063959    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" event={"ID":"b254d093-f40a-4d3d-ab73-7e3c4694a5cd","Type":"ContainerDied","Data":"515241df376288aec8d92357c12746bd9a71874684e895298da227e674bf3eca"}
Oct 31 00:25:45.064326 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.063985    6033 scope.go:122] "RemoveContainer" containerID="809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"
Oct 31 00:25:45.075790 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.075759    6033 scope.go:122] "RemoveContainer" containerID="809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"
Oct 31 00:25:45.076182 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:25:45.076157    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad\": container with ID starting with 809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad not found: ID does not exist" containerID="809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"
Oct 31 00:25:45.076265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.076195    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad"} err="failed to get container status \"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad\": rpc error: code = NotFound desc = could not find container \"809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad\": container with ID starting with 809f430e0fcec7660fa23cfcd369a4ba78b5cb561b7b1eb3928e176b90cc12ad not found: ID does not exist"
Oct 31 00:25:45.220011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.219971    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:25:45.266538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.266495    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4"]
Oct 31 00:25:45.830021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:45.829979    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" path="/var/lib/kubelet/pods/b254d093-f40a-4d3d-ab73-7e3c4694a5cd/volumes"
Oct 31 00:25:46.971277 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:25:46.971237    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:25:47.357126 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:25:47.357081    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ae3cd0-5c94d78dd5-gqds4" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.143:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:05.608439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:05.608396    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:26:05.710318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:05.710281    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:26:10.609811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.609659    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" podUID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" containerName="payload-processing" containerID="cri-o://7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e" gracePeriod=25
Oct 31 00:26:10.710761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.710702    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" podUID="c4386868-b38b-4fe2-990a-8a77e857562f" containerName="payload-pre-processing" containerID="cri-o://15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82" gracePeriod=25
Oct 31 00:26:10.876021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.875947    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:26:10.981438 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.981405    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\" (UniqueName: \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\") pod \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") "
Oct 31 00:26:10.981612 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.981600    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\") pod \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\" (UID: \"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39\") "
Oct 31 00:26:10.981974 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.981948    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume" pod "0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" (UID: "0bc3fe15-87dc-4c09-baf9-0a4e41c12e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:10.983778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.983745    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4" pod "0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" (UID: "0bc3fe15-87dc-4c09-baf9-0a4e41c12e39"). InnerVolumeSpecName "kube-api-access-7c8v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:10.990314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:10.990286    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:26:11.082628 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.082589    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\") pod \"c4386868-b38b-4fe2-990a-8a77e857562f\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") "
Oct 31 00:26:11.082821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.082735    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\" (UniqueName: \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\") pod \"c4386868-b38b-4fe2-990a-8a77e857562f\" (UID: \"c4386868-b38b-4fe2-990a-8a77e857562f\") "
Oct 31 00:26:11.083104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.083077    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume" pod "c4386868-b38b-4fe2-990a-8a77e857562f" (UID: "c4386868-b38b-4fe2-990a-8a77e857562f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:11.083341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.083317    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4386868-b38b-4fe2-990a-8a77e857562f-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:11.083341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.083339    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:11.083533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.083350    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8v4\" (UniqueName: \"kubernetes.io/projected/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39-kube-api-access-7c8v4\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:11.085023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.084996    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf" pod "c4386868-b38b-4fe2-990a-8a77e857562f" (UID: "c4386868-b38b-4fe2-990a-8a77e857562f"). InnerVolumeSpecName "kube-api-access-zwsgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:11.185163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.185068    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwsgf\" (UniqueName: \"kubernetes.io/projected/c4386868-b38b-4fe2-990a-8a77e857562f-kube-api-access-zwsgf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:26:11.271140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.271105    6033 generic.go:356] "Generic (PLEG): container finished" podID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" containerID="7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e" exitCode=0
Oct 31 00:26:11.271311 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.271150    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"
Oct 31 00:26:11.271311 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.271168    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" event={"ID":"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39","Type":"ContainerDied","Data":"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"}
Oct 31 00:26:11.271311 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.271223    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn" event={"ID":"0bc3fe15-87dc-4c09-baf9-0a4e41c12e39","Type":"ContainerDied","Data":"9d450294a2adc039959600fc204b8d2a43a239e8b5e4cdcd3278907c65972b5f"}
Oct 31 00:26:11.271311 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.271255    6033 scope.go:122] "RemoveContainer" containerID="7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"
Oct 31 00:26:11.272877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.272852    6033 generic.go:356] "Generic (PLEG): container finished" podID="c4386868-b38b-4fe2-990a-8a77e857562f" containerID="15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82" exitCode=0
Oct 31 00:26:11.272998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.272904    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"
Oct 31 00:26:11.272998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.272917    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" event={"ID":"c4386868-b38b-4fe2-990a-8a77e857562f","Type":"ContainerDied","Data":"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"}
Oct 31 00:26:11.272998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.272963    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9" event={"ID":"c4386868-b38b-4fe2-990a-8a77e857562f","Type":"ContainerDied","Data":"752b0be1cf362056d3b28a4aa7709f666db5ca441d1f373f767be196ce09849e"}
Oct 31 00:26:11.282279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.282232    6033 scope.go:122] "RemoveContainer" containerID="7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"
Oct 31 00:26:11.282709 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:26:11.282633    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e\": container with ID starting with 7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e not found: ID does not exist" containerID="7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"
Oct 31 00:26:11.282709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.282682    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e"} err="failed to get container status \"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e\": rpc error: code = NotFound desc = could not find container \"7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e\": container with ID starting with 7f56187271e5ced3928064cb84965b0d388ed02517d0a5d674ea736ce957521e not found: ID does not exist"
Oct 31 00:26:11.282709 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.282710    6033 scope.go:122] "RemoveContainer" containerID="15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"
Oct 31 00:26:11.293145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.293089    6033 scope.go:122] "RemoveContainer" containerID="15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"
Oct 31 00:26:11.293482 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:26:11.293453    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82\": container with ID starting with 15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82 not found: ID does not exist" containerID="15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"
Oct 31 00:26:11.293592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.293489    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82"} err="failed to get container status \"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82\": rpc error: code = NotFound desc = could not find container \"15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82\": container with ID starting with 15c5272ca1bb9c7286a811bc7a7483f793784cd4df020666047b511d07b57e82 not found: ID does not exist"
Oct 31 00:26:11.427910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.427872    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:26:11.430583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.430546    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1e5c33-78944dcc56-xzctn"]
Oct 31 00:26:11.532309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.532266    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:26:11.538469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.538433    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1e5c33-56fbff6867-ckkk9"]
Oct 31 00:26:11.831914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.831826    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" path="/var/lib/kubelet/pods/0bc3fe15-87dc-4c09-baf9-0a4e41c12e39/volumes"
Oct 31 00:26:11.832350 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:11.832333    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4386868-b38b-4fe2-990a-8a77e857562f" path="/var/lib/kubelet/pods/c4386868-b38b-4fe2-990a-8a77e857562f/volumes"
Oct 31 00:26:15.053755 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.053716    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"]
Oct 31 00:26:15.100742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.100702    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"]
Oct 31 00:26:15.102955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.102940    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32150c-zl6rx"]
Oct 31 00:26:15.105001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.104971    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1957ba97-hg7n6"]
Oct 31 00:26:15.830619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.830582    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965ae132-1d4b-421d-8c95-626cb485cfce" path="/var/lib/kubelet/pods/965ae132-1d4b-421d-8c95-626cb485cfce/volumes"
Oct 31 00:26:15.831088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:15.831073    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb091328-a1ee-4d24-94c6-04abd70923a3" path="/var/lib/kubelet/pods/bb091328-a1ee-4d24-94c6-04abd70923a3/volumes"
Oct 31 00:26:16.971835 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:26:16.971799    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:26:19.729856 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:19.729803    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:26:29.644218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:29.644183    6033 scope.go:122] "RemoveContainer" containerID="4885432e1131277af6841d99b4d58ce43d38f716df2f46fa9f619519a37c44aa"
Oct 31 00:26:46.970224 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:26:46.970186    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:26:57.370355 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:57.370263    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:26:57.370798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:26:57.370493    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy" containerID="cri-o://54afce903555db05b83f8bc2bf3640a034697cd719db9af61be4e08b3908531d" gracePeriod=30
Oct 31 00:27:01.364755 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:01.364715    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:27:02.626918 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.626886    6033 generic.go:356] "Generic (PLEG): container finished" podID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerID="54afce903555db05b83f8bc2bf3640a034697cd719db9af61be4e08b3908531d" exitCode=0
Oct 31 00:27:02.627361 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.626925    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" event={"ID":"150c932b-2bbd-4454-92d0-dcddf8a4ddea","Type":"ContainerDied","Data":"54afce903555db05b83f8bc2bf3640a034697cd719db9af61be4e08b3908531d"}
Oct 31 00:27:02.657803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.657772    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:27:02.725593 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725557    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.725836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725619    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.725836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725705    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.725836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725734    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.725836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725767    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.725836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725804    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725857    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725911    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.725924    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.726022    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\") pod \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\" (UID: \"150c932b-2bbd-4454-92d0-dcddf8a4ddea\") "
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.726077    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.726129    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.726168    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.726407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.726193    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.727247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.727223    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.727331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.727256    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.727331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.727271    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.727331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.727285    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.727331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.727299    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.728181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.728162    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:02.728299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.728280    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:02.728496 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.728480    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.728548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.728530    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd" pod "150c932b-2bbd-4454-92d0-dcddf8a4ddea" (UID: "150c932b-2bbd-4454-92d0-dcddf8a4ddea"). InnerVolumeSpecName "kube-api-access-lfftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:02.828960 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.828919    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.828960 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.828962    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.829231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.828976    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:02.829231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:02.828990    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/150c932b-2bbd-4454-92d0-dcddf8a4ddea-kube-api-access-lfftd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:03.634551 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:03.634508    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp" event={"ID":"150c932b-2bbd-4454-92d0-dcddf8a4ddea","Type":"ContainerDied","Data":"972d963b7511c90cb0e1e78636d6e7d5afbedb4f522f86f333699539d74885ba"}
Oct 31 00:27:03.635014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:03.634565    6033 scope.go:122] "RemoveContainer" containerID="54afce903555db05b83f8bc2bf3640a034697cd719db9af61be4e08b3908531d"
Oct 31 00:27:03.635014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:03.634525    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"
Oct 31 00:27:03.837860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:03.837819    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:27:03.884692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:03.884593    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-gtzrp"]
Oct 31 00:27:05.829533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:05.829487    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" path="/var/lib/kubelet/pods/150c932b-2bbd-4454-92d0-dcddf8a4ddea/volumes"
Oct 31 00:27:15.056142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.056100    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"]
Oct 31 00:27:15.103167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.103123    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"]
Oct 31 00:27:15.116068 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.116011    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eed3c84-mzvhd"]
Oct 31 00:27:15.116369 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.116348    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-431ac0f9-scd55"]
Oct 31 00:27:15.829491 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.829446    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49b150ef-b32d-4dcc-abb1-da37a970d023" path="/var/lib/kubelet/pods/49b150ef-b32d-4dcc-abb1-da37a970d023/volumes"
Oct 31 00:27:15.829930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:15.829915    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5451011c-f57a-4c3e-9d82-455788d97bec" path="/var/lib/kubelet/pods/5451011c-f57a-4c3e-9d82-455788d97bec/volumes"
Oct 31 00:27:16.970061 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:16.969994    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:27:18.040121 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:18.040078    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 31 00:27:24.959117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:24.959078    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 31 00:27:25.060199 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:25.060155    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 31 00:27:27.841266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:27.841231    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:27:29.688746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.688706    6033 scope.go:122] "RemoveContainer" containerID="a45aa59c3a060754b9ab39a322d029ecb00cca20c701c0e6aba9c03486c9dcde"
Oct 31 00:27:29.699167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.699144    6033 scope.go:122] "RemoveContainer" containerID="3966a13fc4c9831350b6e1b9975affd835f58c9d1c741637ac849196a187fe59"
Oct 31 00:27:29.710052 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.710009    6033 scope.go:122] "RemoveContainer" containerID="c25eab86f4b5090231a58f081ce6fc0c23bb6996094f9b499cacb1c4f3c0b227"
Oct 31 00:27:29.719954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.719924    6033 scope.go:122] "RemoveContainer" containerID="56f434d2beea260f51feca173bf0cccc11a932715d26df468ff72b9bf2c21efc"
Oct 31 00:27:29.730465 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.730335    6033 scope.go:122] "RemoveContainer" containerID="89c060a968f291ffbab49cac43faadd72cb83f7d408e708b1fec226b1a9c290b"
Oct 31 00:27:29.960238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:29.960109    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" podUID="c0181a2c-cd33-450a-a2cc-c59030161f7b" containerName="payload-processing" containerID="cri-o://ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518" gracePeriod=25
Oct 31 00:27:30.060842 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.060774    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" podUID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" containerName="payload-pre-processing" containerID="cri-o://2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2" gracePeriod=25
Oct 31 00:27:30.251976 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.251946    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 31 00:27:30.360344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.360313    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 31 00:27:30.374056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.373993    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\" (UniqueName: \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\") pod \"c0181a2c-cd33-450a-a2cc-c59030161f7b\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") "
Oct 31 00:27:30.374257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.374179    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\") pod \"c0181a2c-cd33-450a-a2cc-c59030161f7b\" (UID: \"c0181a2c-cd33-450a-a2cc-c59030161f7b\") "
Oct 31 00:27:30.374820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.374786    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume" pod "c0181a2c-cd33-450a-a2cc-c59030161f7b" (UID: "c0181a2c-cd33-450a-a2cc-c59030161f7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:30.378466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.378433    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks" pod "c0181a2c-cd33-450a-a2cc-c59030161f7b" (UID: "c0181a2c-cd33-450a-a2cc-c59030161f7b"). InnerVolumeSpecName "kube-api-access-b7gks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:30.476366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.476220    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\" (UniqueName: \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\") pod \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") "
Oct 31 00:27:30.476574 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.476547    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\") pod \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\" (UID: \"8d815df5-b09e-41ff-9eb9-8fc6684dc238\") "
Oct 31 00:27:30.477065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.477020    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume" pod "8d815df5-b09e-41ff-9eb9-8fc6684dc238" (UID: "8d815df5-b09e-41ff-9eb9-8fc6684dc238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:30.477245 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.477227    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d815df5-b09e-41ff-9eb9-8fc6684dc238-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:30.477306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.477251    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7gks\" (UniqueName: \"kubernetes.io/projected/c0181a2c-cd33-450a-a2cc-c59030161f7b-kube-api-access-b7gks\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:30.477306 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.477270    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0181a2c-cd33-450a-a2cc-c59030161f7b-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:30.478964 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.478939    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm" pod "8d815df5-b09e-41ff-9eb9-8fc6684dc238" (UID: "8d815df5-b09e-41ff-9eb9-8fc6684dc238"). InnerVolumeSpecName "kube-api-access-qmvdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:30.578200 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.578158    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmvdm\" (UniqueName: \"kubernetes.io/projected/8d815df5-b09e-41ff-9eb9-8fc6684dc238-kube-api-access-qmvdm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:30.826587 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.826547    6033 generic.go:356] "Generic (PLEG): container finished" podID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" containerID="2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2" exitCode=0
Oct 31 00:27:30.827089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.826600    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"
Oct 31 00:27:30.827089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.826631    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" event={"ID":"8d815df5-b09e-41ff-9eb9-8fc6684dc238","Type":"ContainerDied","Data":"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"}
Oct 31 00:27:30.827089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.826690    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg" event={"ID":"8d815df5-b09e-41ff-9eb9-8fc6684dc238","Type":"ContainerDied","Data":"fbfaad911bbf8d72e4b762aaddaa3096dcfdb603f38b26790fa918f88ad4b886"}
Oct 31 00:27:30.827089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.826717    6033 scope.go:122] "RemoveContainer" containerID="2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"
Oct 31 00:27:30.828229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.828041    6033 generic.go:356] "Generic (PLEG): container finished" podID="c0181a2c-cd33-450a-a2cc-c59030161f7b" containerID="ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518" exitCode=0
Oct 31 00:27:30.828229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.828108    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" event={"ID":"c0181a2c-cd33-450a-a2cc-c59030161f7b","Type":"ContainerDied","Data":"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"}
Oct 31 00:27:30.828229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.828119    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"
Oct 31 00:27:30.828229 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.828145    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff" event={"ID":"c0181a2c-cd33-450a-a2cc-c59030161f7b","Type":"ContainerDied","Data":"663886dfff63d78bac2f1f45dce72461e8e14b14d1de8ce47a60cadba31cd2ce"}
Oct 31 00:27:30.839231 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.839207    6033 scope.go:122] "RemoveContainer" containerID="2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"
Oct 31 00:27:30.839661 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:30.839624    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2\": container with ID starting with 2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2 not found: ID does not exist" containerID="2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"
Oct 31 00:27:30.839780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.839676    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2"} err="failed to get container status \"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2\": rpc error: code = NotFound desc = could not find container \"2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2\": container with ID starting with 2e7f533e60b4a119e1ad62e6a6b379b5bfd8468e7f8345f061673da55f98dbb2 not found: ID does not exist"
Oct 31 00:27:30.839780 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.839703    6033 scope.go:122] "RemoveContainer" containerID="ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"
Oct 31 00:27:30.849482 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.849459    6033 scope.go:122] "RemoveContainer" containerID="ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"
Oct 31 00:27:30.849833 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:30.849810    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518\": container with ID starting with ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518 not found: ID does not exist" containerID="ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"
Oct 31 00:27:30.849911 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.849840    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518"} err="failed to get container status \"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518\": rpc error: code = NotFound desc = could not find container \"ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518\": container with ID starting with ca53596facb1a25e4a932b9b3da04c57beaff045ebebaff4d7158a9e20fc6518 not found: ID does not exist"
Oct 31 00:27:30.981662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:30.981622    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 31 00:27:31.028928 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:31.028891    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-34f1ce-9b77f84c9-jfnmg"]
Oct 31 00:27:31.086681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:31.086591    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 31 00:27:31.095138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:31.095107    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-34f1ce-56d77c46fd-9wsff"]
Oct 31 00:27:31.830530 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:31.830491    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" path="/var/lib/kubelet/pods/8d815df5-b09e-41ff-9eb9-8fc6684dc238/volumes"
Oct 31 00:27:31.831081 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:31.831060    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0181a2c-cd33-450a-a2cc-c59030161f7b" path="/var/lib/kubelet/pods/c0181a2c-cd33-450a-a2cc-c59030161f7b/volumes"
Oct 31 00:27:33.065072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.064994    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" containerID="cri-o://82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be" gracePeriod=45
Oct 31 00:27:33.192649 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.192607    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 00:27:33.336559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.336528    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:27:33.413664 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413623    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413774    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413866    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413895    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\" (UniqueName: \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413999 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413934    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413999 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413963    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\") pod \"8ce9f007-07d1-4172-819b-27b14cecc0a7\" (UID: \"8ce9f007-07d1-4172-819b-27b14cecc0a7\") "
Oct 31 00:27:33.413999 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.413985    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.414212 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.414114    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.414300 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.414274    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.414457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.414440    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.414518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.414481    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.414518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.414497    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.416380 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.416354    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:33.416502 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.416419    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "kube-api-access-7qjsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:33.416502 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.416425    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm" pod "8ce9f007-07d1-4172-819b-27b14cecc0a7" (UID: "8ce9f007-07d1-4172-819b-27b14cecc0a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:33.515893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.515852    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ce9f007-07d1-4172-819b-27b14cecc0a7-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.515893 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.515892    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qjsn\" (UniqueName: \"kubernetes.io/projected/8ce9f007-07d1-4172-819b-27b14cecc0a7-kube-api-access-7qjsn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.516152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.515906    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ce9f007-07d1-4172-819b-27b14cecc0a7-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:33.851281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.851242    6033 generic.go:356] "Generic (PLEG): container finished" podID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerID="82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be" exitCode=0
Oct 31 00:27:33.851473 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.851349    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"
Oct 31 00:27:33.851473 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.851373    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" event={"ID":"8ce9f007-07d1-4172-819b-27b14cecc0a7","Type":"ContainerDied","Data":"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"}
Oct 31 00:27:33.851473 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.851415    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh" event={"ID":"8ce9f007-07d1-4172-819b-27b14cecc0a7","Type":"ContainerDied","Data":"d6591d70ef6323bef796c27c5b6023388ec695e835bbce1f19766d6ef786c31c"}
Oct 31 00:27:33.851473 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.851437    6033 scope.go:122] "RemoveContainer" containerID="82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"
Oct 31 00:27:33.861416 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.861388    6033 scope.go:122] "RemoveContainer" containerID="82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"
Oct 31 00:27:33.861765 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:33.861745    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be\": container with ID starting with 82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be not found: ID does not exist" containerID="82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"
Oct 31 00:27:33.861837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:33.861779    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be"} err="failed to get container status \"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be\": rpc error: code = NotFound desc = could not find container \"82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be\": container with ID starting with 82daf7b4a80ae2e83f7d2b1e71cfeeb1eb3de36406054774c687c1cdc41603be not found: ID does not exist"
Oct 31 00:27:34.005910 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:34.005872    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 31 00:27:34.010810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:34.010782    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-34f1ce/distinct-w4-34f1ce-kserve-85744459ff-7sqnh"]
Oct 31 00:27:35.829989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:35.829948    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" path="/var/lib/kubelet/pods/8ce9f007-07d1-4172-819b-27b14cecc0a7/volumes"
Oct 31 00:27:38.271017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:38.270968    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:27:42.862525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:42.862454    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" podUID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" containerName="main" containerID="cri-o://2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0" gracePeriod=45
Oct 31 00:27:43.153081 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.153048    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:27:43.234991 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.234952    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\" (UniqueName: \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235091    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235132    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235175    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\" (UniqueName: \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235234    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235344    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\") pod \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\" (UID: \"f562d09b-d562-4c4a-8aaf-dcd67f0ce452\") "
Oct 31 00:27:43.235488 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235450    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:43.235540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235468    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:43.235807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235780    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:43.235941 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235901    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.235941 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235927    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.236080 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.235941    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.237688 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.237657    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:43.237794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.237688    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "kube-api-access-vpzn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:43.237794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.237697    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs" pod "f562d09b-d562-4c4a-8aaf-dcd67f0ce452" (UID: "f562d09b-d562-4c4a-8aaf-dcd67f0ce452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:27:43.337550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.337505    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.337550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.337541    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzn4\" (UniqueName: \"kubernetes.io/projected/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-kube-api-access-vpzn4\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.337550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.337552    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f562d09b-d562-4c4a-8aaf-dcd67f0ce452-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:43.916085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.915932    6033 generic.go:356] "Generic (PLEG): container finished" podID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" containerID="2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0" exitCode=0
Oct 31 00:27:43.916085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.915981    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"
Oct 31 00:27:43.916085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.916010    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" event={"ID":"f562d09b-d562-4c4a-8aaf-dcd67f0ce452","Type":"ContainerDied","Data":"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"}
Oct 31 00:27:43.916085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.916091    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs" event={"ID":"f562d09b-d562-4c4a-8aaf-dcd67f0ce452","Type":"ContainerDied","Data":"9e5bb8aa2e439c033372e2dd668e343447026840394c6d6eec2fe501322bbb72"}
Oct 31 00:27:43.916749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.916119    6033 scope.go:122] "RemoveContainer" containerID="2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"
Oct 31 00:27:43.926346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.926320    6033 scope.go:122] "RemoveContainer" containerID="2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"
Oct 31 00:27:43.926781 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:43.926756    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0\": container with ID starting with 2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0 not found: ID does not exist" containerID="2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"
Oct 31 00:27:43.926885 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:43.926792    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0"} err="failed to get container status \"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0\": rpc error: code = NotFound desc = could not find container \"2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0\": container with ID starting with 2b92936467b613a106770c5e6c48d09b00be8db3421727de991e2f7c69c2a0a0 not found: ID does not exist"
Oct 31 00:27:44.069972 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:44.069930    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:27:44.075572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:44.075538    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-76b073/test-model-76b073-kserve-74b8cc9c7c-rp6qs"]
Oct 31 00:27:45.832873 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:45.832833    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" path="/var/lib/kubelet/pods/f562d09b-d562-4c4a-8aaf-dcd67f0ce452/volumes"
Oct 31 00:27:46.971476 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:46.971433    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:27:51.548178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:51.548142    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:27:51.548626 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:51.548384    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" containerID="cri-o://147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036" gracePeriod=30
Oct 31 00:27:54.272427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:54.272386    6033 patch_prober.go:29] interesting pod/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:27:54.272836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:54.272456    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:27:56.830372 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.830342    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:27:56.894656 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894621    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.894860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894662    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.894860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894699    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.894860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894737    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.894860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894802    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.894860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894846    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.895232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894883    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.895232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.894974    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.895232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895080    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\") pod \"9fb1b044-c66c-42f8-884f-665fa0639fab\" (UID: \"9fb1b044-c66c-42f8-884f-665fa0639fab\") "
Oct 31 00:27:56.895232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895107    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:56.895232 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895136    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:27:56.895481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895230    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:56.895481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895287    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:56.895579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895484    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:56.895711 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895691    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.895813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895717    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fb1b044-c66c-42f8-884f-665fa0639fab-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.895813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895732    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.895813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895746    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.895813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.895760    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.897293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.897261    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:27:56.897293 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.897281    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:56.897500 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.897468    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:27:56.897697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.897676    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q" pod "9fb1b044-c66c-42f8-884f-665fa0639fab" (UID: "9fb1b044-c66c-42f8-884f-665fa0639fab"). InnerVolumeSpecName "kube-api-access-8tx4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:27:56.996686 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.996649    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.996686 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.996689    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tx4q\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-kube-api-access-8tx4q\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.996981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.996708    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:56.996981 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:56.996722    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fb1b044-c66c-42f8-884f-665fa0639fab-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:27:57.003759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.003717    6033 generic.go:356] "Generic (PLEG): container finished" podID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerID="147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036" exitCode=0
Oct 31 00:27:57.003946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.003908    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" event={"ID":"9fb1b044-c66c-42f8-884f-665fa0639fab","Type":"ContainerDied","Data":"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"}
Oct 31 00:27:57.004021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.004005    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2" event={"ID":"9fb1b044-c66c-42f8-884f-665fa0639fab","Type":"ContainerDied","Data":"654a3b521cd9792204c3bc442b263446de3fe724de19bed6c0a66bbeb00723f4"}
Oct 31 00:27:57.004094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.004060    6033 scope.go:122] "RemoveContainer" containerID="147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"
Oct 31 00:27:57.004206 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.004182    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"
Oct 31 00:27:57.016089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.016063    6033 scope.go:122] "RemoveContainer" containerID="147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"
Oct 31 00:27:57.016483 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:27:57.016457    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036\": container with ID starting with 147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036 not found: ID does not exist" containerID="147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"
Oct 31 00:27:57.016598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.016490    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036"} err="failed to get container status \"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036\": rpc error: code = NotFound desc = could not find container \"147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036\": container with ID starting with 147fc7f7d8c59bfb00cb33fde96cd569bef70b9d1361217ae0b338d9d47ea036 not found: ID does not exist"
Oct 31 00:27:57.161680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.161573    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:27:57.165746 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.165711    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-34f1ce-openshift-default-856fb7b5bf-m8pb2"]
Oct 31 00:27:57.195680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.195637    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:27:57.829663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.829618    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" path="/var/lib/kubelet/pods/9fb1b044-c66c-42f8-884f-665fa0639fab/volumes"
Oct 31 00:27:57.929008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:27:57.928967    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:28:12.217837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.217792    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main" containerID="cri-o://29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98" gracePeriod=45
Oct 31 00:28:12.492663 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.492629    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:28:12.582187 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582152    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582198    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582264    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\" (UniqueName: \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582290    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582310    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582339    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\") pod \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\" (UID: \"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a\") "
Oct 31 00:28:12.582695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582564    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.582695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582666    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.582695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582682    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.582839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582779    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.582839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582792    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.582839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.582805    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.584725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.584695    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "kube-api-access-88fw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:12.584725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.584701    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:12.584888 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.584710    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs" pod "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" (UID: "0ea89ff8-bb44-4645-9b72-5fb32f0ee31a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:12.684100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.684056    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.684100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.684103    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88fw2\" (UniqueName: \"kubernetes.io/projected/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-kube-api-access-88fw2\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.684318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.684119    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:12.950106 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:12.950064    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" podUID="2ebe9a06-4625-43ea-b41b-15f419f8d292" containerName="main" containerID="cri-o://29533d084c12de91b6fdf863a5edc32dad36ba17f67948564621a17151977e28" gracePeriod=45
Oct 31 00:28:13.109631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.109596    6033 generic.go:356] "Generic (PLEG): container finished" podID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerID="29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98" exitCode=0
Oct 31 00:28:13.109812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.109641    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"
Oct 31 00:28:13.109812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.109683    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" event={"ID":"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a","Type":"ContainerDied","Data":"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"}
Oct 31 00:28:13.109812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.109731    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98" event={"ID":"0ea89ff8-bb44-4645-9b72-5fb32f0ee31a","Type":"ContainerDied","Data":"0847babf33636ea5326feca870d5f9bfcd5eb8f37ea270d8184b60b40d3132a2"}
Oct 31 00:28:13.109812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.109759    6033 scope.go:122] "RemoveContainer" containerID="29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"
Oct 31 00:28:13.112210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.112176    6033 generic.go:356] "Generic (PLEG): container finished" podID="2ebe9a06-4625-43ea-b41b-15f419f8d292" containerID="29533d084c12de91b6fdf863a5edc32dad36ba17f67948564621a17151977e28" exitCode=0
Oct 31 00:28:13.112340 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.112245    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" event={"ID":"2ebe9a06-4625-43ea-b41b-15f419f8d292","Type":"ContainerDied","Data":"29533d084c12de91b6fdf863a5edc32dad36ba17f67948564621a17151977e28"}
Oct 31 00:28:13.122965 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.122939    6033 scope.go:122] "RemoveContainer" containerID="29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"
Oct 31 00:28:13.123526 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:28:13.123465    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98\": container with ID starting with 29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98 not found: ID does not exist" containerID="29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"
Oct 31 00:28:13.123631 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.123538    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98"} err="failed to get container status \"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98\": rpc error: code = NotFound desc = could not find container \"29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98\": container with ID starting with 29de3cd3a575387c5346ca5e3e88319e36d7af17d75d981ef330739593ce9a98 not found: ID does not exist"
Oct 31 00:28:13.208443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.208357    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:28:13.270657 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.270617    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:28:13.291417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291374    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\" (UniqueName: \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291421    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291469    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291519    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291617    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291670    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\") pod \"2ebe9a06-4625-43ea-b41b-15f419f8d292\" (UID: \"2ebe9a06-4625-43ea-b41b-15f419f8d292\") "
Oct 31 00:28:13.291905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291853    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:13.292003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291901    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:13.292003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.291887    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:13.292167 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.292143    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.292210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.292172    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.292210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.292186    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.294045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.293999    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "kube-api-access-9wv6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:13.294045 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.294022    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:13.294201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.294020    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs" pod "2ebe9a06-4625-43ea-b41b-15f419f8d292" (UID: "2ebe9a06-4625-43ea-b41b-15f419f8d292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:13.317681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.317635    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-986d37/distinct-2-w3-986d37-kserve-64c85654c6-t7s98"]
Oct 31 00:28:13.393415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.393370    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebe9a06-4625-43ea-b41b-15f419f8d292-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.393415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.393409    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wv6z\" (UniqueName: \"kubernetes.io/projected/2ebe9a06-4625-43ea-b41b-15f419f8d292-kube-api-access-9wv6z\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.393415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.393420    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebe9a06-4625-43ea-b41b-15f419f8d292-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:13.831235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:13.831197    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" path="/var/lib/kubelet/pods/0ea89ff8-bb44-4645-9b72-5fb32f0ee31a/volumes"
Oct 31 00:28:14.119953 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:14.119858    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"
Oct 31 00:28:14.120478 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:14.120451    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6" event={"ID":"2ebe9a06-4625-43ea-b41b-15f419f8d292","Type":"ContainerDied","Data":"050e2d2d6fc68dd279d8805ea6ad52e55392b1a4a07f1151173a9ee451d16480"}
Oct 31 00:28:14.120596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:14.120502    6033 scope.go:122] "RemoveContainer" containerID="29533d084c12de91b6fdf863a5edc32dad36ba17f67948564621a17151977e28"
Oct 31 00:28:14.276011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:14.275972    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:28:14.322456 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:14.322414    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-49837c/test-model-49837c-kserve-59b78788cf-kvqx6"]
Oct 31 00:28:15.828740 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:15.828705    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ebe9a06-4625-43ea-b41b-15f419f8d292" path="/var/lib/kubelet/pods/2ebe9a06-4625-43ea-b41b-15f419f8d292/volumes"
Oct 31 00:28:16.971021 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:28:16.970983    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:28:29.785774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:29.785740    6033 scope.go:122] "RemoveContainer" containerID="63e6e063ed2aead17a9f85502ed264c6fc76c53e79a5de2d5aa4c2df36e601bb"
Oct 31 00:28:29.795199 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:29.795177    6033 scope.go:122] "RemoveContainer" containerID="e446e952aea98d12ec643d9a8331e68f2fe1f18106832f2a1ea3ad5f1f98839e"
Oct 31 00:28:29.803940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:29.803917    6033 scope.go:122] "RemoveContainer" containerID="6eb51e2f9adeaa20419c1dc7a48d5f06157fb89c8a8a50fb076ff0647b8d2b2d"
Oct 31 00:28:37.307201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:37.307160    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:28:39.009360 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:39.009319    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:28:39.106858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:39.106766    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:28:43.279040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:43.278991    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:28:43.279643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:43.279229    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy" containerID="cri-o://eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27" gracePeriod=30
Oct 31 00:28:44.010007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.009955    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" podUID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" containerName="payload-processing" containerID="cri-o://4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f" gracePeriod=25
Oct 31 00:28:44.107897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.107846    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing" containerID="cri-o://9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c" gracePeriod=25
Oct 31 00:28:44.278982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.278905    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:28:44.316362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.316321    6033 generic.go:356] "Generic (PLEG): container finished" podID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" containerID="4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f" exitCode=0
Oct 31 00:28:44.316847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.316359    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" event={"ID":"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc","Type":"ContainerDied","Data":"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"}
Oct 31 00:28:44.316847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.316400    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"
Oct 31 00:28:44.316847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.316426    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9" event={"ID":"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc","Type":"ContainerDied","Data":"41cfd0af3410fd323416b0d0912535ba4ec89dd7034efe3ac3447447710e586b"}
Oct 31 00:28:44.316847 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.316453    6033 scope.go:122] "RemoveContainer" containerID="4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"
Oct 31 00:28:44.326498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.326470    6033 scope.go:122] "RemoveContainer" containerID="4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"
Oct 31 00:28:44.326865 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:28:44.326834    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f\": container with ID starting with 4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f not found: ID does not exist" containerID="4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"
Oct 31 00:28:44.326969 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.326877    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f"} err="failed to get container status \"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f\": rpc error: code = NotFound desc = could not find container \"4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f\": container with ID starting with 4fe765d960e16f27ec4ff7ed10bf68cfbdf3846ec241328e849073faac94795f not found: ID does not exist"
Oct 31 00:28:44.450444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.450402    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\" (UniqueName: \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\") pod \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") "
Oct 31 00:28:44.450643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.450575    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\") pod \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\" (UID: \"6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc\") "
Oct 31 00:28:44.450960 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.450931    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume" pod "6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" (UID: "6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:44.452674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.452646    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds" pod "6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" (UID: "6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc"). InnerVolumeSpecName "kube-api-access-5jxds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:44.551924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.551824    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:44.551924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.551861    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxds\" (UniqueName: \"kubernetes.io/projected/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc-kube-api-access-5jxds\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:44.730335 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.730293    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:28:44.735344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:44.735308    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49837c-79c55dc878-vd2k9"]
Oct 31 00:28:45.829337 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:45.829295    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" path="/var/lib/kubelet/pods/6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc/volumes"
Oct 31 00:28:46.971992 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:28:46.971950    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:28:47.579598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:47.579550    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:28:48.551725 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.551647    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:28:48.600747 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.600710    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.600949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.600753    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.600949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.600782    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.600949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.600864    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.600999    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601077    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601088    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.601159 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601120    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.601383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601144    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601171    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.601383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601270    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601317    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\") pod \"8a5d6279-2168-4c9d-907e-287eb1965948\" (UID: \"8a5d6279-2168-4c9d-907e-287eb1965948\") "
Oct 31 00:28:48.601620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601398    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:48.601620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601580    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.601862 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601832    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.601998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601865    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.601998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601880    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.601998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601896    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d6279-2168-4c9d-907e-287eb1965948-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.601998 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.601910    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.603280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.603253    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "kube-api-access-skqgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:48.603280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.603260    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:28:48.603425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.603333    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:28:48.603425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.603388    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token" pod "8a5d6279-2168-4c9d-907e-287eb1965948" (UID: "8a5d6279-2168-4c9d-907e-287eb1965948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:48.703013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.702978    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.703013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.703009    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skqgx\" (UniqueName: \"kubernetes.io/projected/8a5d6279-2168-4c9d-907e-287eb1965948-kube-api-access-skqgx\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.703013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.703019    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a5d6279-2168-4c9d-907e-287eb1965948-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:48.703013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:48.703068    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a5d6279-2168-4c9d-907e-287eb1965948-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:49.348619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.348583    6033 generic.go:356] "Generic (PLEG): container finished" podID="8a5d6279-2168-4c9d-907e-287eb1965948" containerID="eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27" exitCode=0
Oct 31 00:28:49.348807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.348634    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"
Oct 31 00:28:49.348807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.348663    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" event={"ID":"8a5d6279-2168-4c9d-907e-287eb1965948","Type":"ContainerDied","Data":"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"}
Oct 31 00:28:49.348807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.348721    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk" event={"ID":"8a5d6279-2168-4c9d-907e-287eb1965948","Type":"ContainerDied","Data":"2be89070768e03a26b2db062278e8ba9dd1ae8ec218c0e9be3bfe831650c95f5"}
Oct 31 00:28:49.348807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.348745    6033 scope.go:122] "RemoveContainer" containerID="eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"
Oct 31 00:28:49.359367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.359343    6033 scope.go:122] "RemoveContainer" containerID="eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"
Oct 31 00:28:49.359704 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:28:49.359681    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27\": container with ID starting with eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27 not found: ID does not exist" containerID="eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"
Oct 31 00:28:49.359775 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.359722    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27"} err="failed to get container status \"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27\": rpc error: code = NotFound desc = could not find container \"eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27\": container with ID starting with eb51a7cf9c9c50b6266e1c1323c08cef23b8e62f4028e9d90b5f0c7f03819b27 not found: ID does not exist"
Oct 31 00:28:49.553788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.553751    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:28:49.600982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.600891    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-nn2nk"]
Oct 31 00:28:49.829813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:49.829768    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" path="/var/lib/kubelet/pods/8a5d6279-2168-4c9d-907e-287eb1965948/volumes"
Oct 31 00:28:57.579086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:57.579012    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:28:58.197279 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.197244    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 31 00:28:58.197481 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.197446    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" podUID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" containerName="maas-api" containerID="cri-o://0eaec6ec55ba8cece4fb7b2eb0b175c03e29b0c28c06f0577a42f5f69ba455f6" gracePeriod=30
Oct 31 00:28:58.416155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.416110    6033 generic.go:356] "Generic (PLEG): container finished" podID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" containerID="0eaec6ec55ba8cece4fb7b2eb0b175c03e29b0c28c06f0577a42f5f69ba455f6" exitCode=0
Oct 31 00:28:58.416448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.416290    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" event={"ID":"f7566f75-16ce-4d60-9361-ccb4f2ef3c28","Type":"ContainerDied","Data":"0eaec6ec55ba8cece4fb7b2eb0b175c03e29b0c28c06f0577a42f5f69ba455f6"}
Oct 31 00:28:58.487546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.487507    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 31 00:28:58.616142 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616106    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\") pod \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") "
Oct 31 00:28:58.616562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616240    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\") pod \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") "
Oct 31 00:28:58.616562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616314    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") pod \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") "
Oct 31 00:28:58.616562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616422    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\" (UniqueName: \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\") pod \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\" (UID: \"f7566f75-16ce-4d60-9361-ccb4f2ef3c28\") "
Oct 31 00:28:58.616731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616549    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca" pod "f7566f75-16ce-4d60-9361-ccb4f2ef3c28" (UID: "f7566f75-16ce-4d60-9361-ccb4f2ef3c28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:28:58.616856 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.616837    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:58.618655 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.618615    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs" pod "f7566f75-16ce-4d60-9361-ccb4f2ef3c28" (UID: "f7566f75-16ce-4d60-9361-ccb4f2ef3c28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:58.618801 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.618652    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls" pod "f7566f75-16ce-4d60-9361-ccb4f2ef3c28" (UID: "f7566f75-16ce-4d60-9361-ccb4f2ef3c28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:28:58.618930 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.618913    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn" pod "f7566f75-16ce-4d60-9361-ccb4f2ef3c28" (UID: "f7566f75-16ce-4d60-9361-ccb4f2ef3c28"). InnerVolumeSpecName "kube-api-access-4lcxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:28:58.718216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.718120    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:58.718216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.718153    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:58.718216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:58.718164    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcxn\" (UniqueName: \"kubernetes.io/projected/f7566f75-16ce-4d60-9361-ccb4f2ef3c28-kube-api-access-4lcxn\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:28:59.423492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.423454    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"
Oct 31 00:28:59.423672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.423455    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2" event={"ID":"f7566f75-16ce-4d60-9361-ccb4f2ef3c28","Type":"ContainerDied","Data":"ac01a232104b306b6dc0e7962372f12aabeb7a50cc182a065305fb481ad57ac4"}
Oct 31 00:28:59.423672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.423649    6033 scope.go:122] "RemoveContainer" containerID="0eaec6ec55ba8cece4fb7b2eb0b175c03e29b0c28c06f0577a42f5f69ba455f6"
Oct 31 00:28:59.578210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.578170    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 31 00:28:59.625002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.624950    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-986d37-fb4fb44ff-cvbh2"]
Oct 31 00:28:59.828759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:28:59.828720    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" path="/var/lib/kubelet/pods/f7566f75-16ce-4d60-9361-ccb4f2ef3c28/volumes"
Oct 31 00:29:04.101417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.101384    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:29:04.101913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.101602    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" podUID="860cf407-971d-4195-b15e-72331da6d9c2" containerName="maas-api" containerID="cri-o://e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197" gracePeriod=30
Oct 31 00:29:04.186795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.186753    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" podUID="860cf407-971d-4195-b15e-72331da6d9c2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.158:8443/health\": dial tcp 10.134.0.158:8443: connect: connection refused"
Oct 31 00:29:04.401642 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.401609    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:29:04.434717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.434688    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.457067    6033 generic.go:356] "Generic (PLEG): container finished" podID="860cf407-971d-4195-b15e-72331da6d9c2" containerID="e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197" exitCode=0
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.457232    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" event={"ID":"860cf407-971d-4195-b15e-72331da6d9c2","Type":"ContainerDied","Data":"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"}
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.457316    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm" event={"ID":"860cf407-971d-4195-b15e-72331da6d9c2","Type":"ContainerDied","Data":"2d54ed4158ee7d56722a0d9c1c94b30c8dd7bbf63c815fe016bf7a9b1b3bbfd8"}
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.457350    6033 scope.go:122] "RemoveContainer" containerID="e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.457553    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.460207    6033 generic.go:356] "Generic (PLEG): container finished" podID="c4a14481-121f-4f88-8305-050e25db4f56" containerID="9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c" exitCode=0
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.460269    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" event={"ID":"c4a14481-121f-4f88-8305-050e25db4f56","Type":"ContainerDied","Data":"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"}
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.460326    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5" event={"ID":"c4a14481-121f-4f88-8305-050e25db4f56","Type":"ContainerDied","Data":"35a67d3a2e2fc3585145c30c4deaf7151df42ff258a69f96abe8f866db4e106e"}
Oct 31 00:29:04.463164 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.460450    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"
Oct 31 00:29:04.472506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.472377    6033 scope.go:122] "RemoveContainer" containerID="e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"
Oct 31 00:29:04.472930 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:29:04.472895    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197\": container with ID starting with e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197 not found: ID does not exist" containerID="e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"
Oct 31 00:29:04.473000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.472943    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197"} err="failed to get container status \"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197\": rpc error: code = NotFound desc = could not find container \"e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197\": container with ID starting with e5940866206e67511f950635181da36ba83e54fbf23f161ecd0665ab20003197 not found: ID does not exist"
Oct 31 00:29:04.473000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.472968    6033 scope.go:122] "RemoveContainer" containerID="9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"
Oct 31 00:29:04.484210 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.484177    6033 scope.go:122] "RemoveContainer" containerID="9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"
Oct 31 00:29:04.484610 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:29:04.484581    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c\": container with ID starting with 9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c not found: ID does not exist" containerID="9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"
Oct 31 00:29:04.484696 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.484624    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c"} err="failed to get container status \"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c\": rpc error: code = NotFound desc = could not find container \"9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c\": container with ID starting with 9b2148868179431b6fb2f20acb4d70f5bf184560c6c42bc0a4cb0f454564751c not found: ID does not exist"
Oct 31 00:29:04.485912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.485888    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\") pod \"860cf407-971d-4195-b15e-72331da6d9c2\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") "
Oct 31 00:29:04.486011 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.485995    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") pod \"860cf407-971d-4195-b15e-72331da6d9c2\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") "
Oct 31 00:29:04.486182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.486164    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\" (UniqueName: \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\") pod \"860cf407-971d-4195-b15e-72331da6d9c2\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") "
Oct 31 00:29:04.486255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.486237    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\") pod \"860cf407-971d-4195-b15e-72331da6d9c2\" (UID: \"860cf407-971d-4195-b15e-72331da6d9c2\") "
Oct 31 00:29:04.486366 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.486345    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca" pod "860cf407-971d-4195-b15e-72331da6d9c2" (UID: "860cf407-971d-4195-b15e-72331da6d9c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:04.486603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.486582    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860cf407-971d-4195-b15e-72331da6d9c2-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.488215 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.488189    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls" pod "860cf407-971d-4195-b15e-72331da6d9c2" (UID: "860cf407-971d-4195-b15e-72331da6d9c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:04.488325 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.488287    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs" pod "860cf407-971d-4195-b15e-72331da6d9c2" (UID: "860cf407-971d-4195-b15e-72331da6d9c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:29:04.488373 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.488328    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd" pod "860cf407-971d-4195-b15e-72331da6d9c2" (UID: "860cf407-971d-4195-b15e-72331da6d9c2"). InnerVolumeSpecName "kube-api-access-9dftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:04.587762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.587721    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\") pod \"c4a14481-121f-4f88-8305-050e25db4f56\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") "
Oct 31 00:29:04.587962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.587790    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\" (UniqueName: \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\") pod \"c4a14481-121f-4f88-8305-050e25db4f56\" (UID: \"c4a14481-121f-4f88-8305-050e25db4f56\") "
Oct 31 00:29:04.588084 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.588065    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.588151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.588089    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dftd\" (UniqueName: \"kubernetes.io/projected/860cf407-971d-4195-b15e-72331da6d9c2-kube-api-access-9dftd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.588151 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.588102    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860cf407-971d-4195-b15e-72331da6d9c2-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.588255 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.588178    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume" pod "c4a14481-121f-4f88-8305-050e25db4f56" (UID: "c4a14481-121f-4f88-8305-050e25db4f56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:29:04.590209 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.590174    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2" pod "c4a14481-121f-4f88-8305-050e25db4f56" (UID: "c4a14481-121f-4f88-8305-050e25db4f56"). InnerVolumeSpecName "kube-api-access-98kr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:29:04.688935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.688835    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a14481-121f-4f88-8305-050e25db4f56-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.688935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.688874    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98kr2\" (UniqueName: \"kubernetes.io/projected/c4a14481-121f-4f88-8305-050e25db4f56-kube-api-access-98kr2\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:29:04.875503 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.875465    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:29:04.879890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.879858    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49837c-76f4b86dff-x4tvm"]
Oct 31 00:29:04.988474 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.988381    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:29:04.992796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:04.992760    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49837c-5c557bd847-pn6w5"]
Oct 31 00:29:05.829996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:05.829945    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860cf407-971d-4195-b15e-72331da6d9c2" path="/var/lib/kubelet/pods/860cf407-971d-4195-b15e-72331da6d9c2/volumes"
Oct 31 00:29:05.830611 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:05.830585    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a14481-121f-4f88-8305-050e25db4f56" path="/var/lib/kubelet/pods/c4a14481-121f-4f88-8305-050e25db4f56/volumes"
Oct 31 00:29:16.198812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:16.198769    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:29:16.970836 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:29:16.970788    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:29:29.868607 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:29.868568    6033 scope.go:122] "RemoveContainer" containerID="68b83541606662a8f01cd942ef5d9b734bd4a027716fb85617fb8eea1ee4949c"
Oct 31 00:29:46.636024 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:29:46.635974    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:29:46.971488 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:29:46.971382    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:30:00.166459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166419    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:00.166935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166906    6033 cpu_manager.go:414] "containerMap: removing container" podUID="860cf407-971d-4195-b15e-72331da6d9c2" containerName="maas-api"
Oct 31 00:30:00.166935 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166917    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860cf407-971d-4195-b15e-72331da6d9c2" containerName="maas-api"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166939    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" containerName="main"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166944    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" containerName="main"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166954    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c4386868-b38b-4fe2-990a-8a77e857562f" containerName="payload-pre-processing"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166960    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4386868-b38b-4fe2-990a-8a77e857562f" containerName="payload-pre-processing"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166970    6033 cpu_manager.go:414] "containerMap: removing container" podUID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" containerName="payload-pre-processing"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166975    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" containerName="payload-pre-processing"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166983    6033 cpu_manager.go:414] "containerMap: removing container" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166989    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.166998    6033 cpu_manager.go:414] "containerMap: removing container" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167002    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167008    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy"
Oct 31 00:30:00.167008 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167013    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167022    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167041    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167056    6033 cpu_manager.go:414] "containerMap: removing container" podUID="2ebe9a06-4625-43ea-b41b-15f419f8d292" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167060    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ebe9a06-4625-43ea-b41b-15f419f8d292" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167068    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167072    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167080    6033 cpu_manager.go:414] "containerMap: removing container" podUID="c0181a2c-cd33-450a-a2cc-c59030161f7b" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167085    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0181a2c-cd33-450a-a2cc-c59030161f7b" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167091    6033 cpu_manager.go:414] "containerMap: removing container" podUID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167096    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167104    6033 cpu_manager.go:414] "containerMap: removing container" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167108    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167116    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167121    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167126    6033 cpu_manager.go:414] "containerMap: removing container" podUID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167131    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167138    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167142    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167156    6033 cpu_manager.go:414] "containerMap: removing container" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167160    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167166    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167171    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167241    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="150c932b-2bbd-4454-92d0-dcddf8a4ddea" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167248    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5d6279-2168-4c9d-907e-287eb1965948" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167255    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d815df5-b09e-41ff-9eb9-8fc6684dc238" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167264    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ebe9a06-4625-43ea-b41b-15f419f8d292" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167270    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a14481-121f-4f88-8305-050e25db4f56" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167276    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0181a2c-cd33-450a-a2cc-c59030161f7b" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167282    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7566f75-16ce-4d60-9361-ccb4f2ef3c28" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167287    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb1b044-c66c-42f8-884f-665fa0639fab" containerName="istio-proxy"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167292    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad36225a-ea11-4d9a-8efd-8e917933bc5f" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167297    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc3fe15-87dc-4c09-baf9-0a4e41c12e39" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167304    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce9f007-07d1-4172-819b-27b14cecc0a7" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167310    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="b254d093-f40a-4d3d-ab73-7e3c4694a5cd" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167315    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa0b10b-2d81-44dd-bd1f-3e9eee121bfc" containerName="payload-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167321    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="860cf407-971d-4195-b15e-72331da6d9c2" containerName="maas-api"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167328    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="f562d09b-d562-4c4a-8aaf-dcd67f0ce452" containerName="main"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167334    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4386868-b38b-4fe2-990a-8a77e857562f" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167339    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9783b1-d7b7-44a1-8bef-8a323435aee2" containerName="payload-pre-processing"
Oct 31 00:30:00.167414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.167344    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea89ff8-bb44-4645-9b72-5fb32f0ee31a" containerName="main"
Oct 31 00:30:00.170758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.170734    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.223813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.223780    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:30:00.272086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.272014    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:00.311489 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.311447    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgq52\" (UniqueName: \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.311719 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.311577    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.412744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.412697    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.412943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.412846    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgq52\" (UniqueName: \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.413439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.413418    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.471906 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.471827    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgq52\" (UniqueName: \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\") pod \"maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.483671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.483634    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:00.858289 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:00.858262    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:00.861158 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:30:00.861129    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3489c442_5217_458e_a6ba_84bec2326a7d.slice/crio-8c7f38c55252ee046fb3f6f0dd0c9977ffac68fc6e3767657418a294aba22223 WatchSource:0}: Error finding container 8c7f38c55252ee046fb3f6f0dd0c9977ffac68fc6e3767657418a294aba22223: Status 404 returned error can't find the container with id 8c7f38c55252ee046fb3f6f0dd0c9977ffac68fc6e3767657418a294aba22223
Oct 31 00:30:01.782742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:01.782714    6033 generic.go:356] "Generic (PLEG): container finished" podID="3489c442-5217-458e-a6ba-84bec2326a7d" containerID="1d5399e173c8023191c8f2c1c5072ff9a8dd5bb1e256c7b0e1ce77084281da71" exitCode=60
Oct 31 00:30:01.783117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:01.782775    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" event={"ID":"3489c442-5217-458e-a6ba-84bec2326a7d","Type":"ContainerDied","Data":"1d5399e173c8023191c8f2c1c5072ff9a8dd5bb1e256c7b0e1ce77084281da71"}
Oct 31 00:30:01.783117 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:01.782818    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" event={"ID":"3489c442-5217-458e-a6ba-84bec2326a7d","Type":"ContainerStarted","Data":"8c7f38c55252ee046fb3f6f0dd0c9977ffac68fc6e3767657418a294aba22223"}
Oct 31 00:30:01.783205 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:01.783185    6033 scope.go:122] "RemoveContainer" containerID="1d5399e173c8023191c8f2c1c5072ff9a8dd5bb1e256c7b0e1ce77084281da71"
Oct 31 00:30:02.258913 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.258864    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"]
Oct 31 00:30:02.263112 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.263087    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.317617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.317581    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"]
Oct 31 00:30:02.318207 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.318169    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:02.334812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.334760    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwp9\" (UniqueName: \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.335051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.334878    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.335051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.334914    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.435745 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.435700    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwp9\" (UniqueName: \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.435947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.435759    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.435947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.435781    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.436528 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.436504    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.438364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.438344    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.496113 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.496081    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwp9\" (UniqueName: \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\") pod \"maas-api-revoke-sub-0441080d-gwzts\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.576733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.576636    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:02.790364 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.790328    6033 generic.go:356] "Generic (PLEG): container finished" podID="3489c442-5217-458e-a6ba-84bec2326a7d" containerID="52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656" exitCode=60
Oct 31 00:30:02.790802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.790410    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" event={"ID":"3489c442-5217-458e-a6ba-84bec2326a7d","Type":"ContainerDied","Data":"52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656"}
Oct 31 00:30:02.790802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.790474    6033 scope.go:122] "RemoveContainer" containerID="1d5399e173c8023191c8f2c1c5072ff9a8dd5bb1e256c7b0e1ce77084281da71"
Oct 31 00:30:02.790802 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.790763    6033 scope.go:122] "RemoveContainer" containerID="52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656"
Oct 31 00:30:02.791102 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:02.791082    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d_odh-ai-gateway-infra(3489c442-5217-458e-a6ba-84bec2326a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" podUID="3489c442-5217-458e-a6ba-84bec2326a7d"
Oct 31 00:30:02.947315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:02.947228    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"]
Oct 31 00:30:02.950503 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:30:02.950462    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfc757e_27e7_4dde_ae21_c60549fe3c10.slice/crio-d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1 WatchSource:0}: Error finding container d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1: Status 404 returned error can't find the container with id d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1
Oct 31 00:30:03.798441 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:03.798409    6033 scope.go:122] "RemoveContainer" containerID="52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656"
Oct 31 00:30:03.798946 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:03.798710    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d_odh-ai-gateway-infra(3489c442-5217-458e-a6ba-84bec2326a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" podUID="3489c442-5217-458e-a6ba-84bec2326a7d"
Oct 31 00:30:03.799816 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:03.799786    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts" event={"ID":"dcfc757e-27e7-4dde-ae21-c60549fe3c10","Type":"ContainerStarted","Data":"358db2022674d77299a57ac1fc75f28f503be1a058f5a40c0e55bf64381bb252"}
Oct 31 00:30:03.799922 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:03.799841    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts" event={"ID":"dcfc757e-27e7-4dde-ae21-c60549fe3c10","Type":"ContainerStarted","Data":"d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1"}
Oct 31 00:30:04.016511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:04.016451    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts" podStartSLOduration=2.016435956 podStartE2EDuration="2.016435956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:04.01553148 +0000 UTC m=+2856.362515378" watchObservedRunningTime="2026-10-31 00:30:04.016435956 +0000 UTC m=+2856.363419854"
Oct 31 00:30:04.807505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:04.807469    6033 generic.go:356] "Generic (PLEG): container finished" podID="dcfc757e-27e7-4dde-ae21-c60549fe3c10" containerID="358db2022674d77299a57ac1fc75f28f503be1a058f5a40c0e55bf64381bb252" exitCode=0
Oct 31 00:30:04.808004 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:04.807553    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts" event={"ID":"dcfc757e-27e7-4dde-ae21-c60549fe3c10","Type":"ContainerDied","Data":"358db2022674d77299a57ac1fc75f28f503be1a058f5a40c0e55bf64381bb252"}
Oct 31 00:30:05.955272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:05.955242    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:06.076256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.076153    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\") pod \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") "
Oct 31 00:30:06.076435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.076331    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\" (UniqueName: \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\") pod \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") "
Oct 31 00:30:06.076435 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.076378    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\") pod \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\" (UID: \"dcfc757e-27e7-4dde-ae21-c60549fe3c10\") "
Oct 31 00:30:06.077104 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.077063    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca" pod "dcfc757e-27e7-4dde-ae21-c60549fe3c10" (UID: "dcfc757e-27e7-4dde-ae21-c60549fe3c10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:06.079594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.079563    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca" pod "dcfc757e-27e7-4dde-ae21-c60549fe3c10" (UID: "dcfc757e-27e7-4dde-ae21-c60549fe3c10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:06.079881 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.079814    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9" pod "dcfc757e-27e7-4dde-ae21-c60549fe3c10" (UID: "dcfc757e-27e7-4dde-ae21-c60549fe3c10"). InnerVolumeSpecName "kube-api-access-4kwp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:06.178168 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.178135    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.178353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.178181    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwp9\" (UniqueName: \"kubernetes.io/projected/dcfc757e-27e7-4dde-ae21-c60549fe3c10-kube-api-access-4kwp9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.178353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.178193    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfc757e-27e7-4dde-ae21-c60549fe3c10-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:06.819759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.819717    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts" event={"ID":"dcfc757e-27e7-4dde-ae21-c60549fe3c10","Type":"ContainerDied","Data":"d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1"}
Oct 31 00:30:06.819968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.819770    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5cf260608e12c1e5ce40cf21dbb95443495c4f5680196c6fb8a04e62ed8bea1"
Oct 31 00:30:06.819968 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:06.819805    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"
Oct 31 00:30:14.556065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:14.556001    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:30:15.499759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.499723    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"]
Oct 31 00:30:15.500247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.500233    6033 cpu_manager.go:414] "containerMap: removing container" podUID="dcfc757e-27e7-4dde-ae21-c60549fe3c10" containerName="revoke-keys"
Oct 31 00:30:15.500299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.500248    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfc757e-27e7-4dde-ae21-c60549fe3c10" containerName="revoke-keys"
Oct 31 00:30:15.500383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.500373    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfc757e-27e7-4dde-ae21-c60549fe3c10" containerName="revoke-keys"
Oct 31 00:30:15.504367 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.504339    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.558099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.558064    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"]
Oct 31 00:30:15.602140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.602102    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:30:15.683528 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.683490    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.683752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.683552    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlbc\" (UniqueName: \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.683752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.683735    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.784538 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.784500    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.784778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.784562    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.784778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.784614    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlbc\" (UniqueName: \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.785303 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.785274    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.787138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.787108    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:15.842668 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:15.842636    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlbc\" (UniqueName: \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\") pod \"maas-api-revoke-sub-e220b258-k7j4q\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:16.119283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.119195    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:16.492892 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.492865    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"]
Oct 31 00:30:16.495638 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:30:16.495601    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f0cb5e_d1e4_41a1_a63f_4e9d4ebc7ec9.slice/crio-afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9 WatchSource:0}: Error finding container afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9: Status 404 returned error can't find the container with id afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9
Oct 31 00:30:16.822218 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.822181    6033 scope.go:122] "RemoveContainer" containerID="52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656"
Oct 31 00:30:16.876527 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.876487    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q" event={"ID":"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9","Type":"ContainerStarted","Data":"e061a4496444506bbb25c7c7f8eed7a5861cb3d6d311ea0c081b068a1a5d670f"}
Oct 31 00:30:16.876695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.876551    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q" event={"ID":"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9","Type":"ContainerStarted","Data":"afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9"}
Oct 31 00:30:16.967181 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.967148    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:30:16.973268 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:16.973227    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:30:16.988698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:16.987568    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q" podStartSLOduration=1.987546689 podStartE2EDuration="1.987546689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:16.985719919 +0000 UTC m=+2869.332703818" watchObservedRunningTime="2026-10-31 00:30:16.987546689 +0000 UTC m=+2869.334530588"
Oct 31 00:30:17.884579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.884491    6033 generic.go:356] "Generic (PLEG): container finished" podID="3489c442-5217-458e-a6ba-84bec2326a7d" containerID="82d1eac2f0394e31934214597b8d47fe1cca9a8f236dce0fadd66397940232c0" exitCode=60
Oct 31 00:30:17.884579 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.884561    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" event={"ID":"3489c442-5217-458e-a6ba-84bec2326a7d","Type":"ContainerDied","Data":"82d1eac2f0394e31934214597b8d47fe1cca9a8f236dce0fadd66397940232c0"}
Oct 31 00:30:17.885101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.884626    6033 scope.go:122] "RemoveContainer" containerID="52ea12fa49427c32b0570a1b230e0f41802925cc8d52af8504dd99a7ee6d0656"
Oct 31 00:30:17.885101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.885070    6033 scope.go:122] "RemoveContainer" containerID="82d1eac2f0394e31934214597b8d47fe1cca9a8f236dce0fadd66397940232c0"
Oct 31 00:30:17.885399 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:17.885372    6033 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d_odh-ai-gateway-infra(3489c442-5217-458e-a6ba-84bec2326a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" podUID="3489c442-5217-458e-a6ba-84bec2326a7d"
Oct 31 00:30:17.886322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.886295    6033 generic.go:356] "Generic (PLEG): container finished" podID="90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" containerID="e061a4496444506bbb25c7c7f8eed7a5861cb3d6d311ea0c081b068a1a5d670f" exitCode=0
Oct 31 00:30:17.886404 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:17.886366    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q" event={"ID":"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9","Type":"ContainerDied","Data":"e061a4496444506bbb25c7c7f8eed7a5861cb3d6d311ea0c081b068a1a5d670f"}
Oct 31 00:30:19.008072 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.008011    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:19.040253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.040215    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:19.125599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.124622    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\") pod \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") "
Oct 31 00:30:19.125599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.124696    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\" (UniqueName: \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\") pod \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") "
Oct 31 00:30:19.125599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.125100    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\") pod \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\" (UID: \"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9\") "
Oct 31 00:30:19.125599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.125201    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca" pod "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" (UID: "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:19.125599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.125560    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.128417 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.128378    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca" pod "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" (UID: "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:19.131670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.131637    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc" pod "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" (UID: "90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9"). InnerVolumeSpecName "kube-api-access-7hlbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:19.168040 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.167999    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:19.226165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226129    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\" (UniqueName: \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\") pod \"3489c442-5217-458e-a6ba-84bec2326a7d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") "
Oct 31 00:30:19.226324 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226268    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\") pod \"3489c442-5217-458e-a6ba-84bec2326a7d\" (UID: \"3489c442-5217-458e-a6ba-84bec2326a7d\") "
Oct 31 00:30:19.226630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226608    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca" pod "3489c442-5217-458e-a6ba-84bec2326a7d" (UID: "3489c442-5217-458e-a6ba-84bec2326a7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:19.226758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226738    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlbc\" (UniqueName: \"kubernetes.io/projected/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-kube-api-access-7hlbc\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.226795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226765    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3489c442-5217-458e-a6ba-84bec2326a7d-maas-api-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.226795 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.226780    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.228411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.228388    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52" pod "3489c442-5217-458e-a6ba-84bec2326a7d" (UID: "3489c442-5217-458e-a6ba-84bec2326a7d"). InnerVolumeSpecName "kube-api-access-jgq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:19.328101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.328024    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgq52\" (UniqueName: \"kubernetes.io/projected/3489c442-5217-458e-a6ba-84bec2326a7d-kube-api-access-jgq52\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:19.902971 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.902936    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"
Oct 31 00:30:19.902971 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.902959    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d" event={"ID":"3489c442-5217-458e-a6ba-84bec2326a7d","Type":"ContainerDied","Data":"8c7f38c55252ee046fb3f6f0dd0c9977ffac68fc6e3767657418a294aba22223"}
Oct 31 00:30:19.903275 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.903056    6033 scope.go:122] "RemoveContainer" containerID="82d1eac2f0394e31934214597b8d47fe1cca9a8f236dce0fadd66397940232c0"
Oct 31 00:30:19.904407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.904336    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q" event={"ID":"90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9","Type":"ContainerDied","Data":"afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9"}
Oct 31 00:30:19.904407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.904356    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"
Oct 31 00:30:19.904407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:19.904380    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afcb7840669e3655224802efd70886b0314077abbc432a51a25514a0d43fa9d9"
Oct 31 00:30:20.057317 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:20.057280    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:20.061537 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:20.061497    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-37abc0-29890110-ww66d"]
Oct 31 00:30:21.828419 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:21.828384    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" path="/var/lib/kubelet/pods/3489c442-5217-458e-a6ba-84bec2326a7d/volumes"
Oct 31 00:30:29.576341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.576273    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerName="main" containerID="cri-o://3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352" gracePeriod=45
Oct 31 00:30:29.692211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.692170    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": dial tcp 10.134.0.167:8000: connect: connection refused"
Oct 31 00:30:29.844125 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.844049    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:30:29.947097 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947051    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947110    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947154    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947206    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\" (UniqueName: \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947241    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947278    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\") pod \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\" (UID: \"fac9a388-5d54-453f-bf9d-71aabb9bb90e\") "
Oct 31 00:30:29.947571 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947319    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:29.947571 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947401    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:29.947571 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947524    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:29.947794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947770    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:29.947794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947795    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:29.948013 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.947811    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:29.949720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.949694    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "kube-api-access-t677v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:29.949720 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.949701    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:29.949890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.949788    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs" pod "fac9a388-5d54-453f-bf9d-71aabb9bb90e" (UID: "fac9a388-5d54-453f-bf9d-71aabb9bb90e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:29.962969 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.962935    6033 generic.go:356] "Generic (PLEG): container finished" podID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerID="3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352" exitCode=0
Oct 31 00:30:29.963170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.962991    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"
Oct 31 00:30:29.963170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.963015    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" event={"ID":"fac9a388-5d54-453f-bf9d-71aabb9bb90e","Type":"ContainerDied","Data":"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"}
Oct 31 00:30:29.963170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.963083    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg" event={"ID":"fac9a388-5d54-453f-bf9d-71aabb9bb90e","Type":"ContainerDied","Data":"25d86b6c1d670e17799d51838d65bc47dd91cd729b8039910fa50e312de047ff"}
Oct 31 00:30:29.963170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.963106    6033 scope.go:122] "RemoveContainer" containerID="3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"
Oct 31 00:30:29.976459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.976436    6033 scope.go:122] "RemoveContainer" containerID="3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"
Oct 31 00:30:29.976796 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:29.976773    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352\": container with ID starting with 3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352 not found: ID does not exist" containerID="3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"
Oct 31 00:30:29.976894 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:29.976807    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352"} err="failed to get container status \"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352\": rpc error: code = NotFound desc = could not find container \"3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352\": container with ID starting with 3a1d60074384c34d2a89afd0f069718c136c5313d31698de28ff2e753b767352 not found: ID does not exist"
Oct 31 00:30:30.048599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:30.048553    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac9a388-5d54-453f-bf9d-71aabb9bb90e-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.048599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:30.048592    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t677v\" (UniqueName: \"kubernetes.io/projected/fac9a388-5d54-453f-bf9d-71aabb9bb90e-kube-api-access-t677v\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.048599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:30.048606    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac9a388-5d54-453f-bf9d-71aabb9bb90e-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:30.118876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:30.118787    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:30:30.124360 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:30.124323    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ddf61a/test-model-ddf61a-kserve-5bb9f849c5-9sjqg"]
Oct 31 00:30:31.799125 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:31.799085    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 31 00:30:31.829337 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:31.829305    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" path="/var/lib/kubelet/pods/fac9a388-5d54-453f-bf9d-71aabb9bb90e/volumes"
Oct 31 00:30:31.845105 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:31.845072    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 31 00:30:36.999799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:36.999700    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:30:38.822440 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:38.822401    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 31 00:30:43.822733 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:43.822681    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" podUID="0e62af63-1817-44c7-b273-2ec390b0eb94" containerName="payload-processing" containerID="cri-o://8612d8b3409ab59b442e0242ca79a7317c6dc0fa3c62559a32508cdf38958e90" gracePeriod=25
Oct 31 00:30:44.042398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.042364    6033 generic.go:356] "Generic (PLEG): container finished" podID="0e62af63-1817-44c7-b273-2ec390b0eb94" containerID="8612d8b3409ab59b442e0242ca79a7317c6dc0fa3c62559a32508cdf38958e90" exitCode=0
Oct 31 00:30:44.042596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.042442    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" event={"ID":"0e62af63-1817-44c7-b273-2ec390b0eb94","Type":"ContainerDied","Data":"8612d8b3409ab59b442e0242ca79a7317c6dc0fa3c62559a32508cdf38958e90"}
Oct 31 00:30:44.091380 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.091300    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 31 00:30:44.106377 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.106339    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\") pod \"0e62af63-1817-44c7-b273-2ec390b0eb94\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") "
Oct 31 00:30:44.106572 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.106406    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\" (UniqueName: \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\") pod \"0e62af63-1817-44c7-b273-2ec390b0eb94\" (UID: \"0e62af63-1817-44c7-b273-2ec390b0eb94\") "
Oct 31 00:30:44.106756 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.106732    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume" pod "0e62af63-1817-44c7-b273-2ec390b0eb94" (UID: "0e62af63-1817-44c7-b273-2ec390b0eb94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:30:44.108753 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.108721    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb" pod "0e62af63-1817-44c7-b273-2ec390b0eb94" (UID: "0e62af63-1817-44c7-b273-2ec390b0eb94"). InnerVolumeSpecName "kube-api-access-dsgzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:44.207837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.207793    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e62af63-1817-44c7-b273-2ec390b0eb94-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:44.207837 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:44.207835    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgzb\" (UniqueName: \"kubernetes.io/projected/0e62af63-1817-44c7-b273-2ec390b0eb94-kube-api-access-dsgzb\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:45.050351 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.050315    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"
Oct 31 00:30:45.050855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.050342    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj" event={"ID":"0e62af63-1817-44c7-b273-2ec390b0eb94","Type":"ContainerDied","Data":"778162e5fd8ba84f050fdc64466bd75181ead84eb5798f418a6dcb77cc2fd36f"}
Oct 31 00:30:45.050855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.050416    6033 scope.go:122] "RemoveContainer" containerID="8612d8b3409ab59b442e0242ca79a7317c6dc0fa3c62559a32508cdf38958e90"
Oct 31 00:30:45.205842 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.205802    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 31 00:30:45.252682 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.252643    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-46b586-556df6d97b-dcfpj"]
Oct 31 00:30:45.828820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:45.828774    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e62af63-1817-44c7-b273-2ec390b0eb94" path="/var/lib/kubelet/pods/0e62af63-1817-44c7-b273-2ec390b0eb94/volumes"
Oct 31 00:30:46.820086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:46.819997    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main" containerID="cri-o://3cb804a49653c6bf9a8a775d105ade9ea366a2189adfa711bbf7268eb3f374c5" gracePeriod=45
Oct 31 00:30:46.866444 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:46.866403    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" containerID="cri-o://56e1cbf78f97da54494365625cd8593f9519125600ced31fc1b2e99857a98e35" gracePeriod=45
Oct 31 00:30:46.970816 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:30:46.970778    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:30:47.065942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.065910    6033 generic.go:356] "Generic (PLEG): container finished" podID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerID="56e1cbf78f97da54494365625cd8593f9519125600ced31fc1b2e99857a98e35" exitCode=0
Oct 31 00:30:47.066156 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.065983    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" event={"ID":"d7d6d862-8f30-4ca5-a85e-f78b41912730","Type":"ContainerDied","Data":"56e1cbf78f97da54494365625cd8593f9519125600ced31fc1b2e99857a98e35"}
Oct 31 00:30:47.067680 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.067657    6033 generic.go:356] "Generic (PLEG): container finished" podID="042b4b34-8fc8-4171-a166-78bac0196e01" containerID="3cb804a49653c6bf9a8a775d105ade9ea366a2189adfa711bbf7268eb3f374c5" exitCode=0
Oct 31 00:30:47.067799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.067706    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" event={"ID":"042b4b34-8fc8-4171-a166-78bac0196e01","Type":"ContainerDied","Data":"3cb804a49653c6bf9a8a775d105ade9ea366a2189adfa711bbf7268eb3f374c5"}
Oct 31 00:30:47.160453 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.160421    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:30:47.165539 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.165498    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:30:47.244706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244667    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\" (UniqueName: \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.244887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244721    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\" (UniqueName: \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.244887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244786    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.244887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244821    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.244887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244866    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.245140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244939    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\" (UniqueName: \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.245140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.244997    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.245140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245051    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.245140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245086    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.245140 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245127    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245156    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\") pod \"042b4b34-8fc8-4171-a166-78bac0196e01\" (UID: \"042b4b34-8fc8-4171-a166-78bac0196e01\") "
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245187    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\") pod \"d7d6d862-8f30-4ca5-a85e-f78b41912730\" (UID: \"d7d6d862-8f30-4ca5-a85e-f78b41912730\") "
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.245949    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.246138    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.246269    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.246597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.246361    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.246933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.246633    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.246933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.246706    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.248374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.248336    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:47.248937 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.248912    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "kube-api-access-xzn5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:47.249150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.249116    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.249506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.249375    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs" pod "d7d6d862-8f30-4ca5-a85e-f78b41912730" (UID: "d7d6d862-8f30-4ca5-a85e-f78b41912730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:30:47.249506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.249473    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:30:47.249506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.249480    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8" pod "042b4b34-8fc8-4171-a166-78bac0196e01" (UID: "042b4b34-8fc8-4171-a166-78bac0196e01"). InnerVolumeSpecName "kube-api-access-mvtl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346838    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346874    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d862-8f30-4ca5-a85e-f78b41912730-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346884    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346906    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042b4b34-8fc8-4171-a166-78bac0196e01-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346916    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346924    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346931    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346939    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346948    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042b4b34-8fc8-4171-a166-78bac0196e01-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.346947 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346956    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7d6d862-8f30-4ca5-a85e-f78b41912730-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.347744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346971    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzn5d\" (UniqueName: \"kubernetes.io/projected/d7d6d862-8f30-4ca5-a85e-f78b41912730-kube-api-access-xzn5d\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:47.347744 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:47.346985    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtl8\" (UniqueName: \"kubernetes.io/projected/042b4b34-8fc8-4171-a166-78bac0196e01-kube-api-access-mvtl8\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:30:48.074459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.074423    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"
Oct 31 00:30:48.074955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.074426    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" event={"ID":"d7d6d862-8f30-4ca5-a85e-f78b41912730","Type":"ContainerDied","Data":"c168f1a97f67d3715477a69a64dd7fc889a4ab6e5e58451e242c5285a0f58703"}
Oct 31 00:30:48.074955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.074565    6033 scope.go:122] "RemoveContainer" containerID="56e1cbf78f97da54494365625cd8593f9519125600ced31fc1b2e99857a98e35"
Oct 31 00:30:48.076515 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.076493    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"
Oct 31 00:30:48.076677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.076521    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj" event={"ID":"042b4b34-8fc8-4171-a166-78bac0196e01","Type":"ContainerDied","Data":"1d949b46c47b1673f415e10d4c119ecbbe5a9f1e2f11b995ae187a21539e6b1e"}
Oct 31 00:30:48.086192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.086166    6033 scope.go:122] "RemoveContainer" containerID="3cb804a49653c6bf9a8a775d105ade9ea366a2189adfa711bbf7268eb3f374c5"
Oct 31 00:30:48.119943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.119896    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:30:48.228453 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.228411    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 31 00:30:48.234897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.234854    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/distinct-2-w0-46b586-kserve-6f7d78987-l7xtj"]
Oct 31 00:30:48.337754 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.337654    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 31 00:30:48.342381 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:48.342346    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-46b586/premium-w0-46b586-kserve-66cd484bb9-6wqrp"]
Oct 31 00:30:49.829602 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:49.829567    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" path="/var/lib/kubelet/pods/042b4b34-8fc8-4171-a166-78bac0196e01/volumes"
Oct 31 00:30:49.830022 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:49.830008    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" path="/var/lib/kubelet/pods/d7d6d862-8f30-4ca5-a85e-f78b41912730/volumes"
Oct 31 00:30:59.996679 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:30:59.996642    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:31:02.717710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:02.717660    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:31:02.818073 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:02.818022    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:31:07.718652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:07.718558    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing" containerID="cri-o://300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8" gracePeriod=25
Oct 31 00:31:07.818648 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:07.818578    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing" containerID="cri-o://103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261" gracePeriod=25
Oct 31 00:31:11.595915 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.595869    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:31:11.596434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.596178    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" podUID="1d0b0037-1655-4a52-a525-b72cb0a503dd" containerName="maas-api" containerID="cri-o://be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087" gracePeriod=30
Oct 31 00:31:11.875111 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.875083    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:31:11.921575 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.921532    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") pod \"1d0b0037-1655-4a52-a525-b72cb0a503dd\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") "
Oct 31 00:31:11.921778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.921611    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\") pod \"1d0b0037-1655-4a52-a525-b72cb0a503dd\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") "
Oct 31 00:31:11.921778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.921725    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\") pod \"1d0b0037-1655-4a52-a525-b72cb0a503dd\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") "
Oct 31 00:31:11.921778 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.921771    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\" (UniqueName: \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\") pod \"1d0b0037-1655-4a52-a525-b72cb0a503dd\" (UID: \"1d0b0037-1655-4a52-a525-b72cb0a503dd\") "
Oct 31 00:31:11.922383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.922169    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca" pod "1d0b0037-1655-4a52-a525-b72cb0a503dd" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:11.924067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.924022    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr" pod "1d0b0037-1655-4a52-a525-b72cb0a503dd" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd"). InnerVolumeSpecName "kube-api-access-4xlgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:11.924395 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.924361    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs" pod "1d0b0037-1655-4a52-a525-b72cb0a503dd" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:11.924510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:11.924434    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls" pod "1d0b0037-1655-4a52-a525-b72cb0a503dd" (UID: "1d0b0037-1655-4a52-a525-b72cb0a503dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:12.023605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.023562    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xlgr\" (UniqueName: \"kubernetes.io/projected/1d0b0037-1655-4a52-a525-b72cb0a503dd-kube-api-access-4xlgr\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:12.023605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.023600    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:12.023605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.023610    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d0b0037-1655-4a52-a525-b72cb0a503dd-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:12.023842 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.023619    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b0037-1655-4a52-a525-b72cb0a503dd-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:12.212573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.212469    6033 generic.go:356] "Generic (PLEG): container finished" podID="1d0b0037-1655-4a52-a525-b72cb0a503dd" containerID="be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087" exitCode=0
Oct 31 00:31:12.212573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.212521    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"
Oct 31 00:31:12.212573 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.212552    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" event={"ID":"1d0b0037-1655-4a52-a525-b72cb0a503dd","Type":"ContainerDied","Data":"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"}
Oct 31 00:31:12.212868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.212608    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf" event={"ID":"1d0b0037-1655-4a52-a525-b72cb0a503dd","Type":"ContainerDied","Data":"14487627886ca4cd3c2ae740ae1a2034b2f4107be54498c495494560fd07aef2"}
Oct 31 00:31:12.212868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.212624    6033 scope.go:122] "RemoveContainer" containerID="be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"
Oct 31 00:31:12.222792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.222766    6033 scope.go:122] "RemoveContainer" containerID="be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"
Oct 31 00:31:12.223175 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:12.223143    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087\": container with ID starting with be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087 not found: ID does not exist" containerID="be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"
Oct 31 00:31:12.223292 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.223177    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087"} err="failed to get container status \"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087\": rpc error: code = NotFound desc = could not find container \"be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087\": container with ID starting with be626c098f2c42127a6e634d8d1e36e15b171419908841ef333260c88540a087 not found: ID does not exist"
Oct 31 00:31:12.366591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.366553    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:31:12.371708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:12.371671    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ddf61a-97b755fcd-7c9lf"]
Oct 31 00:31:13.828349 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:13.828313    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0b0037-1655-4a52-a525-b72cb0a503dd" path="/var/lib/kubelet/pods/1d0b0037-1655-4a52-a525-b72cb0a503dd/volumes"
Oct 31 00:31:16.175560 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:16.175515    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:16.175560 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:16.175540    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:16.970491 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:16.970452    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:31:20.017318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:20.017278    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 31 00:31:20.017862 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:20.017552    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" containerID="cri-o://e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93" gracePeriod=30
Oct 31 00:31:22.108295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:22.108258    6033 patch_prober.go:29] interesting pod/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:31:22.108692 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:22.108327    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:31:25.276497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.276468    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 31 00:31:25.284411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.284327    6033 generic.go:356] "Generic (PLEG): container finished" podID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerID="e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93" exitCode=0
Oct 31 00:31:25.284411 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.284374    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"
Oct 31 00:31:25.284596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.284408    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" event={"ID":"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f","Type":"ContainerDied","Data":"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"}
Oct 31 00:31:25.284596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.284465    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98" event={"ID":"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f","Type":"ContainerDied","Data":"8ccd767f3a699fa20b72a9a23332253478e38de381e045d409e1a6a32a8648a2"}
Oct 31 00:31:25.284596 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.284488    6033 scope.go:122] "RemoveContainer" containerID="e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"
Oct 31 00:31:25.294055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.294012    6033 scope.go:122] "RemoveContainer" containerID="e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"
Oct 31 00:31:25.294365 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:25.294343    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93\": container with ID starting with e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93 not found: ID does not exist" containerID="e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"
Oct 31 00:31:25.294412 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.294381    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93"} err="failed to get container status \"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93\": rpc error: code = NotFound desc = could not find container \"e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93\": container with ID starting with e7b8189921ba84653dc0b0d796f311a0499b0775ebd94824a42a76e15e2e5a93 not found: ID does not exist"
Oct 31 00:31:25.372415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372373    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372480    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372523    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372585    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372613    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372647    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372691    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372754    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.372855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372801    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:25.372855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372824    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\") pod \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\" (UID: \"ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f\") "
Oct 31 00:31:25.373165 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.372950    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.373228 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373157    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.373284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373238    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.373284 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373248    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.373402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373339    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.373402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373351    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.373402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373360    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.373402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373370    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.373402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.373378    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.375065 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.375009    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:31:25.375178 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.375104    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "kube-api-access-d4wdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:25.375310 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.375284    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:25.375310 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.375296    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy" pod "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" (UID: "ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:31:25.474320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.474271    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.474320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.474311    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wdh\" (UniqueName: \"kubernetes.io/projected/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-kube-api-access-d4wdh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.474320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.474323    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.474320 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.474332    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:25.699599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.699557    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 31 00:31:25.703007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.702975    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-46b586-openshift-default-86c598c96c-f6k98"]
Oct 31 00:31:25.829418 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:25.829382    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" path="/var/lib/kubelet/pods/ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f/volumes"
Oct 31 00:31:26.176021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:26.175977    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:26.176021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:26.176002    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:27.995445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:27.995410    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:31:28.087292 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.087262    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:31:28.100629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.100587    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\" (UniqueName: \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\") pod \"5029d054-88b7-4c49-8954-0783911fbb4c\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") "
Oct 31 00:31:28.100821 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.100761    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\") pod \"5029d054-88b7-4c49-8954-0783911fbb4c\" (UID: \"5029d054-88b7-4c49-8954-0783911fbb4c\") "
Oct 31 00:31:28.101153 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.101127    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume" pod "5029d054-88b7-4c49-8954-0783911fbb4c" (UID: "5029d054-88b7-4c49-8954-0783911fbb4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:28.102914 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.102892    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh" pod "5029d054-88b7-4c49-8954-0783911fbb4c" (UID: "5029d054-88b7-4c49-8954-0783911fbb4c"). InnerVolumeSpecName "kube-api-access-5xfbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:28.201866 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.201826    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\") pod \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") "
Oct 31 00:31:28.202069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.202017    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\" (UniqueName: \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\") pod \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\" (UID: \"4c29a153-f0a0-406e-8ce3-ba51e3d3a587\") "
Oct 31 00:31:28.202322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.202293    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume" pod "4c29a153-f0a0-406e-8ce3-ba51e3d3a587" (UID: "4c29a153-f0a0-406e-8ce3-ba51e3d3a587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:28.202448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.202367    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.202448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.202384    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfbh\" (UniqueName: \"kubernetes.io/projected/5029d054-88b7-4c49-8954-0783911fbb4c-kube-api-access-5xfbh\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.202448 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.202399    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5029d054-88b7-4c49-8954-0783911fbb4c-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.204118 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.204091    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm" pod "4c29a153-f0a0-406e-8ce3-ba51e3d3a587" (UID: "4c29a153-f0a0-406e-8ce3-ba51e3d3a587"). InnerVolumeSpecName "kube-api-access-tnsvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:28.303471 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.303432    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsvm\" (UniqueName: \"kubernetes.io/projected/4c29a153-f0a0-406e-8ce3-ba51e3d3a587-kube-api-access-tnsvm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:28.304673 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.304643    6033 generic.go:356] "Generic (PLEG): container finished" podID="5029d054-88b7-4c49-8954-0783911fbb4c" containerID="300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8" exitCode=0
Oct 31 00:31:28.304815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.304710    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"
Oct 31 00:31:28.304815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.304735    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" event={"ID":"5029d054-88b7-4c49-8954-0783911fbb4c","Type":"ContainerDied","Data":"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"}
Oct 31 00:31:28.304815 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.304812    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj" event={"ID":"5029d054-88b7-4c49-8954-0783911fbb4c","Type":"ContainerDied","Data":"901f5ff27614cb4966cc0c6f6a84d1eecf62072b24475a5aada44691fbbc7554"}
Oct 31 00:31:28.304939 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.304845    6033 scope.go:122] "RemoveContainer" containerID="300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"
Oct 31 00:31:28.306492 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.306465    6033 generic.go:356] "Generic (PLEG): container finished" podID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerID="103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261" exitCode=0
Oct 31 00:31:28.306617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.306502    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" event={"ID":"4c29a153-f0a0-406e-8ce3-ba51e3d3a587","Type":"ContainerDied","Data":"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"}
Oct 31 00:31:28.306617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.306530    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"
Oct 31 00:31:28.306617 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.306551    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x" event={"ID":"4c29a153-f0a0-406e-8ce3-ba51e3d3a587","Type":"ContainerDied","Data":"acb0aad59195b36a58dddad483d19dfe0a28e367ffcef3a2f438ed7e89e8606e"}
Oct 31 00:31:28.316945 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.316782    6033 scope.go:122] "RemoveContainer" containerID="300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"
Oct 31 00:31:28.317444 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:28.317414    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8\": container with ID starting with 300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8 not found: ID does not exist" containerID="300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"
Oct 31 00:31:28.317578 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.317458    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8"} err="failed to get container status \"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8\": rpc error: code = NotFound desc = could not find container \"300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8\": container with ID starting with 300c911a875745f493722f65dd3eba2df8f80adc299d167c295990eb45802fa8 not found: ID does not exist"
Oct 31 00:31:28.317578 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.317484    6033 scope.go:122] "RemoveContainer" containerID="103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"
Oct 31 00:31:28.328071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.328043    6033 scope.go:122] "RemoveContainer" containerID="103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"
Oct 31 00:31:28.328426 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:28.328402    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261\": container with ID starting with 103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261 not found: ID does not exist" containerID="103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"
Oct 31 00:31:28.328509 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.328440    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261"} err="failed to get container status \"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261\": rpc error: code = NotFound desc = could not find container \"103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261\": container with ID starting with 103096948c4a74f055e51132f9dcfa12bc248ea413c13d959629a89bf27ab261 not found: ID does not exist"
Oct 31 00:31:28.459635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.459589    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:31:28.463838 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.463800    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ddf61a-5f445c7548-fwpmj"]
Oct 31 00:31:28.592251 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.592165    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:31:28.601921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.601880    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:31:28.621948 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:28.621911    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ddf61a-67ff4567bc-mrc2x"]
Oct 31 00:31:29.828600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:29.828566    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" path="/var/lib/kubelet/pods/4c29a153-f0a0-406e-8ce3-ba51e3d3a587/volumes"
Oct 31 00:31:29.828984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:29.828953    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" path="/var/lib/kubelet/pods/5029d054-88b7-4c49-8954-0783911fbb4c/volumes"
Oct 31 00:31:33.730699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:33.730662    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:31:38.731144 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:38.731082    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing" containerID="cri-o://e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b" gracePeriod=25
Oct 31 00:31:46.088781 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.088744    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:31:46.089256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.088969    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" podUID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" containerName="maas-api" containerID="cri-o://6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6" gracePeriod=30
Oct 31 00:31:46.352523 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.352451    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:31:46.400085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.400054    6033 generic.go:356] "Generic (PLEG): container finished" podID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" containerID="6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6" exitCode=0
Oct 31 00:31:46.400085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.400086    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"
Oct 31 00:31:46.400319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.400134    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" event={"ID":"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5","Type":"ContainerDied","Data":"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"}
Oct 31 00:31:46.400319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.400192    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9" event={"ID":"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5","Type":"ContainerDied","Data":"11a8f5eba1d666dde554c4a474ab0ecb25d770779977ade7d5ee54a4c50afa18"}
Oct 31 00:31:46.400319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.400214    6033 scope.go:122] "RemoveContainer" containerID="6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"
Oct 31 00:31:46.411559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.411530    6033 scope.go:122] "RemoveContainer" containerID="6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"
Oct 31 00:31:46.411918 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:46.411888    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6\": container with ID starting with 6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6 not found: ID does not exist" containerID="6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"
Oct 31 00:31:46.412127 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.412096    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6"} err="failed to get container status \"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6\": rpc error: code = NotFound desc = could not find container \"6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6\": container with ID starting with 6a2a3a375f982ed7adb7579954d006b070ea5771f87c2eac0c96a332f1cacdd6 not found: ID does not exist"
Oct 31 00:31:46.490109 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.490075    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\" (UniqueName: \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\") pod \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") "
Oct 31 00:31:46.490376 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.490134    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\") pod \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") "
Oct 31 00:31:46.490376 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.490259    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\") pod \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") "
Oct 31 00:31:46.490376 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.490358    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\") pod \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\" (UID: \"dbc9fd65-4b35-4a84-9c04-64d03ae28fe5\") "
Oct 31 00:31:46.490811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.490783    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca" pod "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" (UID: "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:46.492559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.492535    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls" pod "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" (UID: "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:46.492785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.492762    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n" pod "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" (UID: "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5"). InnerVolumeSpecName "kube-api-access-p5x4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:46.492861 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.492766    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs" pod "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" (UID: "dbc9fd65-4b35-4a84-9c04-64d03ae28fe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:31:46.591535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.591500    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-maas-api-tls\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:46.591535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.591534    6033 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-openshift-service-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:46.591535 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.591544    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5x4n\" (UniqueName: \"kubernetes.io/projected/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-kube-api-access-p5x4n\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:46.591796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.591556    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:46.814078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.814001    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:31:46.816265 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:46.816235    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37abc0-5b8d676757-9f6b9"]
Oct 31 00:31:46.973366 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:46.973277    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:31:47.504987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:47.504945    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:47.828813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:47.828731    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" path="/var/lib/kubelet/pods/dbc9fd65-4b35-4a84-9c04-64d03ae28fe5/volumes"
Oct 31 00:31:51.850077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:51.850022    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:31:57.504737 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:57.504695    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:58.992626 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:58.992537    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:31:59.009347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.009305    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\") pod \"7f4087af-277c-4ee8-a262-73477f30a70b\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") "
Oct 31 00:31:59.009347 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.009353    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\" (UniqueName: \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\") pod \"7f4087af-277c-4ee8-a262-73477f30a70b\" (UID: \"7f4087af-277c-4ee8-a262-73477f30a70b\") "
Oct 31 00:31:59.009822 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.009793    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume" pod "7f4087af-277c-4ee8-a262-73477f30a70b" (UID: "7f4087af-277c-4ee8-a262-73477f30a70b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:59.011689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.011660    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5" pod "7f4087af-277c-4ee8-a262-73477f30a70b" (UID: "7f4087af-277c-4ee8-a262-73477f30a70b"). InnerVolumeSpecName "kube-api-access-44bs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:59.110570 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.110529    6033 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f4087af-277c-4ee8-a262-73477f30a70b-plugins-config-volume\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:59.110570 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.110572    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bs5\" (UniqueName: \"kubernetes.io/projected/7f4087af-277c-4ee8-a262-73477f30a70b-kube-api-access-44bs5\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:31:59.470261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.470222    6033 generic.go:356] "Generic (PLEG): container finished" podID="7f4087af-277c-4ee8-a262-73477f30a70b" containerID="e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b" exitCode=0
Oct 31 00:31:59.470261 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.470271    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"
Oct 31 00:31:59.470562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.470298    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" event={"ID":"7f4087af-277c-4ee8-a262-73477f30a70b","Type":"ContainerDied","Data":"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"}
Oct 31 00:31:59.470562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.470369    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh" event={"ID":"7f4087af-277c-4ee8-a262-73477f30a70b","Type":"ContainerDied","Data":"ba7bf6f2691270ad9015a089e73179ab0e6479bbcc244073f28eb82fe1c49c9a"}
Oct 31 00:31:59.470562 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.470399    6033 scope.go:122] "RemoveContainer" containerID="e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"
Oct 31 00:31:59.481294 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.481262    6033 scope.go:122] "RemoveContainer" containerID="e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"
Oct 31 00:31:59.481540 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:31:59.481502    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b\": container with ID starting with e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b not found: ID does not exist" containerID="e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"
Oct 31 00:31:59.481594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.481556    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b"} err="failed to get container status \"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b\": rpc error: code = NotFound desc = could not find container \"e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b\": container with ID starting with e855f7314592382112a1a958dab705bdf0b0ea852eecc38b7bc8fb311847864b not found: ID does not exist"
Oct 31 00:31:59.632280 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.632242    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:31:59.678946 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.678911    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-37abc0-69469c56c-6bzrh"]
Oct 31 00:31:59.837524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:31:59.837479    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" path="/var/lib/kubelet/pods/7f4087af-277c-4ee8-a262-73477f30a70b/volumes"
Oct 31 00:32:00.539130 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:00.539067    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 31 00:32:00.585304 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:00.585270    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 31 00:32:11.068884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.068786    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"]
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069235    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069248    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069258    6033 cpu_manager.go:414] "containerMap: removing container" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerName="main"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069263    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerName="main"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069276    6033 cpu_manager.go:414] "containerMap: removing container" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069281    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069290    6033 cpu_manager.go:414] "containerMap: removing container" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069295    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069304    6033 cpu_manager.go:414] "containerMap: removing container" podUID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" containerName="maas-api"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069309    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" containerName="maas-api"
Oct 31 00:32:11.069315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069321    6033 cpu_manager.go:414] "containerMap: removing container" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069326    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069335    6033 cpu_manager.go:414] "containerMap: removing container" podUID="90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" containerName="revoke-keys"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069340    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" containerName="revoke-keys"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069348    6033 cpu_manager.go:414] "containerMap: removing container" podUID="1d0b0037-1655-4a52-a525-b72cb0a503dd" containerName="maas-api"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069352    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0b0037-1655-4a52-a525-b72cb0a503dd" containerName="maas-api"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069365    6033 cpu_manager.go:414] "containerMap: removing container" podUID="0e62af63-1817-44c7-b273-2ec390b0eb94" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069370    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e62af63-1817-44c7-b273-2ec390b0eb94" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069377    6033 cpu_manager.go:414] "containerMap: removing container" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069381    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069394    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069398    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069405    6033 cpu_manager.go:414] "containerMap: removing container" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069410    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069419    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069423    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069488    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e62af63-1817-44c7-b273-2ec390b0eb94" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069497    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4087af-277c-4ee8-a262-73477f30a70b" containerName="payload-pre-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069505    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecce1a5a-d4ce-42ea-b2e8-53c43ea33d7f" containerName="istio-proxy"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069519    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="042b4b34-8fc8-4171-a166-78bac0196e01" containerName="main"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069527    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5029d054-88b7-4c49-8954-0783911fbb4c" containerName="payload-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069535    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c29a153-f0a0-406e-8ce3-ba51e3d3a587" containerName="payload-pre-processing"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069541    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac9a388-5d54-453f-bf9d-71aabb9bb90e" containerName="main"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069546    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069552    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d6d862-8f30-4ca5-a85e-f78b41912730" containerName="main"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069558    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc9fd65-4b35-4a84-9c04-64d03ae28fe5" containerName="maas-api"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069562    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069568    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069573    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" containerName="revoke-keys"
Oct 31 00:32:11.069693 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.069578    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0b0037-1655-4a52-a525-b72cb0a503dd" containerName="maas-api"
Oct 31 00:32:11.075609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.074467    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.127586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.127542    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"]
Oct 31 00:32:11.130846 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.130805    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.131056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.130997    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9qq\" (UniqueName: \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.131138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.131113    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.172051 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.171998    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:32:11.172260 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.172076    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:32:11.174621 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.174590    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:32:11.174785 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.174773    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:32:11.232550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.232514    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.232729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.232608    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9qq\" (UniqueName: \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.232729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.232635    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.233463 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.233441    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.234989 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.234969    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.291391 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.291356    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9qq\" (UniqueName: \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\") pod \"maas-api-revoke-sub-3073a8d8-sgh5h\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.390590 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.390500    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:11.760399 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:11.760354    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"]
Oct 31 00:32:11.763161 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:32:11.763121    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d03c369_01a5_4a1f_88a9_48c2d5dd27fd.slice/crio-b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db WatchSource:0}: Error finding container b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db: Status 404 returned error can't find the container with id b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db
Oct 31 00:32:12.541849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:12.541808    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h" event={"ID":"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd","Type":"ContainerStarted","Data":"2a8e68178f29aefca2bde940216867e73d9aaf2575c618cc0503e210d8cbef5c"}
Oct 31 00:32:12.542296 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:12.541875    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h" event={"ID":"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd","Type":"ContainerStarted","Data":"b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db"}
Oct 31 00:32:12.651639 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:12.651583    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h" podStartSLOduration=1.651570301 podStartE2EDuration="1.651570301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:12.650525894 +0000 UTC m=+2984.997509792" watchObservedRunningTime="2026-10-31 00:32:12.651570301 +0000 UTC m=+2984.998554198"
Oct 31 00:32:13.548466 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:13.548435    6033 generic.go:356] "Generic (PLEG): container finished" podID="4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" containerID="2a8e68178f29aefca2bde940216867e73d9aaf2575c618cc0503e210d8cbef5c" exitCode=0
Oct 31 00:32:13.548883 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:13.548502    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h" event={"ID":"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd","Type":"ContainerDied","Data":"2a8e68178f29aefca2bde940216867e73d9aaf2575c618cc0503e210d8cbef5c"}
Oct 31 00:32:14.695101 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.695065    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:14.766525 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.766483    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\" (UniqueName: \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\") pod \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") "
Oct 31 00:32:14.766742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.766590    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\") pod \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") "
Oct 31 00:32:14.766742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.766670    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\") pod \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\" (UID: \"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd\") "
Oct 31 00:32:14.767088 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.767021    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca" pod "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" (UID: "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:14.768658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.768638    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq" pod "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" (UID: "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd"). InnerVolumeSpecName "kube-api-access-vd9qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:14.768752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.768731    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca" pod "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" (UID: "4d03c369-01a5-4a1f-88a9-48c2d5dd27fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:14.868055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.867937    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9qq\" (UniqueName: \"kubernetes.io/projected/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-kube-api-access-vd9qq\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:14.868055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.867976    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:14.868055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:14.867986    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.559722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.559657    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main" containerID="cri-o://4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365" gracePeriod=45
Oct 31 00:32:15.560980 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.560950    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"
Oct 31 00:32:15.561130 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.560990    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h" event={"ID":"4d03c369-01a5-4a1f-88a9-48c2d5dd27fd","Type":"ContainerDied","Data":"b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db"}
Oct 31 00:32:15.561130 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.561048    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b278c5d28cc961017507cc168f3526ecc88a8dae750774b98860a2b9209683db"
Oct 31 00:32:15.605533 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.605481    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main" containerID="cri-o://e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b" gracePeriod=45
Oct 31 00:32:15.860469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.860439    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:32:15.892580 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.892551    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:32:15.978299 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978265    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.978464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978310    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.978464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978345    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.978464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978392    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.978464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978450    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978560    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\" (UniqueName: \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978617    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978634    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978681    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978714    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978747    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.978787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978784    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\") pod \"6bea949e-21bc-4993-8004-9922b5dbd11b\" (UID: \"6bea949e-21bc-4993-8004-9922b5dbd11b\") "
Oct 31 00:32:15.979192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978820    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.979192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978847    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\" (UniqueName: \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.979192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978875    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\") pod \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\" (UID: \"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b\") "
Oct 31 00:32:15.979192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.978930    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.979413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979261    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.979413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979310    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.979413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979330    6033 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-home\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.979413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979344    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.979413 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979359    6033 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-model-cache\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:15.979694 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.979440    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.981597 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.981569    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.981748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.981685    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:15.981748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.981715    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:15.981869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.981815    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:15.981933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.981877    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6" pod "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" (UID: "bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b"). InnerVolumeSpecName "kube-api-access-ttsj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:15.982124 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:15.982101    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd" pod "6bea949e-21bc-4993-8004-9922b5dbd11b" (UID: "6bea949e-21bc-4993-8004-9922b5dbd11b"). InnerVolumeSpecName "kube-api-access-z5tbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:16.080149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080106    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea949e-21bc-4993-8004-9922b5dbd11b-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080141    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5tbd\" (UniqueName: \"kubernetes.io/projected/6bea949e-21bc-4993-8004-9922b5dbd11b-kube-api-access-z5tbd\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080151    6033 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tls-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080172    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080181    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea949e-21bc-4993-8004-9922b5dbd11b-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080190    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttsj6\" (UniqueName: \"kubernetes.io/projected/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-kube-api-access-ttsj6\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080200    6033 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-tmp-dir\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.080414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.080209    6033 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b-dshm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:16.567455 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.567420    6033 generic.go:356] "Generic (PLEG): container finished" podID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerID="4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365" exitCode=0
Oct 31 00:32:16.567618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.567465    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"
Oct 31 00:32:16.567618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.567502    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" event={"ID":"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b","Type":"ContainerDied","Data":"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"}
Oct 31 00:32:16.567618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.567545    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72" event={"ID":"bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b","Type":"ContainerDied","Data":"0e4832d64394ed03fc4d1b722e15d546559539843022942561825a98ead2d105"}
Oct 31 00:32:16.567618 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.567569    6033 scope.go:122] "RemoveContainer" containerID="4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"
Oct 31 00:32:16.568942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.568800    6033 generic.go:356] "Generic (PLEG): container finished" podID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerID="e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b" exitCode=0
Oct 31 00:32:16.568942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.568842    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" event={"ID":"6bea949e-21bc-4993-8004-9922b5dbd11b","Type":"ContainerDied","Data":"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"}
Oct 31 00:32:16.568942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.568858    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"
Oct 31 00:32:16.568942 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.568870    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w" event={"ID":"6bea949e-21bc-4993-8004-9922b5dbd11b","Type":"ContainerDied","Data":"f8f28ab4949a59e79de06bddf5efd9740ffbfa5672db5c5b480e29229fc34363"}
Oct 31 00:32:16.577536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.577515    6033 scope.go:122] "RemoveContainer" containerID="4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"
Oct 31 00:32:16.577856 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:16.577827    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365\": container with ID starting with 4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365 not found: ID does not exist" containerID="4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"
Oct 31 00:32:16.577940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.577868    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365"} err="failed to get container status \"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365\": rpc error: code = NotFound desc = could not find container \"4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365\": container with ID starting with 4ab7a2e2076852a83a9e4e6725c58b8f9786d3c098b7a2ea922d2e9354034365 not found: ID does not exist"
Oct 31 00:32:16.577940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.577893    6033 scope.go:122] "RemoveContainer" containerID="e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"
Oct 31 00:32:16.586216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.586194    6033 scope.go:122] "RemoveContainer" containerID="e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"
Oct 31 00:32:16.586525 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:16.586507    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b\": container with ID starting with e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b not found: ID does not exist" containerID="e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"
Oct 31 00:32:16.586600 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.586534    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b"} err="failed to get container status \"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b\": rpc error: code = NotFound desc = could not find container \"e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b\": container with ID starting with e5d871adcc1255805760796bfdcbb6a42c3b13123cb49aea30f1b5613e0b1d8b not found: ID does not exist"
Oct 31 00:32:16.723498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.723461    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 31 00:32:16.770330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.770295    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/simulator-w1-5c3374-kserve-df759567-nbp9w"]
Oct 31 00:32:16.874451 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.874366    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 31 00:32:16.878889 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:16.878861    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5c3374/distinct-w1-5c3374-kserve-549997c7c7-nsq72"]
Oct 31 00:32:16.968964 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:16.968926    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:32:17.829896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:17.829858    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" path="/var/lib/kubelet/pods/6bea949e-21bc-4993-8004-9922b5dbd11b/volumes"
Oct 31 00:32:17.830359 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:17.830343    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" path="/var/lib/kubelet/pods/bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b/volumes"
Oct 31 00:32:28.038710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:28.038671    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:32:36.353592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:36.353551    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:32:36.354050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:36.353767    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" containerID="cri-o://9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d" gracePeriod=30
Oct 31 00:32:39.679646 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:39.679609    6033 patch_prober.go:29] interesting pod/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:32:39.680081 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:39.679666    6033 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:32:41.609499 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.609466    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:32:41.691796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.691706    6033 generic.go:356] "Generic (PLEG): container finished" podID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerID="9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d" exitCode=0
Oct 31 00:32:41.691796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.691754    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"
Oct 31 00:32:41.691982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.691791    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" event={"ID":"cf36b503-0a11-463a-b78c-368fabfc6d68","Type":"ContainerDied","Data":"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"}
Oct 31 00:32:41.691982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.691872    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm" event={"ID":"cf36b503-0a11-463a-b78c-368fabfc6d68","Type":"ContainerDied","Data":"6d96479573f2d95dbafb75f259a2a56a1d5b1f6bb70232a0cf8c8784c543bae6"}
Oct 31 00:32:41.691982 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.691894    6033 scope.go:122] "RemoveContainer" containerID="9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"
Oct 31 00:32:41.701731 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.701707    6033 scope.go:122] "RemoveContainer" containerID="9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"
Oct 31 00:32:41.702078 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:41.702057    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d\": container with ID starting with 9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d not found: ID does not exist" containerID="9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"
Oct 31 00:32:41.702152 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.702091    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d"} err="failed to get container status \"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d\": rpc error: code = NotFound desc = could not find container \"9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d\": container with ID starting with 9f1319cf3e166663c84ed91b7eb6294d54457f2a29fe8b40fe85411bcc22991d not found: ID does not exist"
Oct 31 00:32:41.721643 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.721606    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 31 00:32:41.721820 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.721793    6033 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager" containerID="cri-o://1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7" gracePeriod=30
Oct 31 00:32:41.728601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728565    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728601 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728605    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728629    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728672    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728792 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728718    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728793    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728843    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.728938 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728877    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.729146 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.728929    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:41.729146 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729105    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\") pod \"cf36b503-0a11-463a-b78c-368fabfc6d68\" (UID: \"cf36b503-0a11-463a-b78c-368fabfc6d68\") "
Oct 31 00:32:41.729146 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729130    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:41.729402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729376    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:41.729734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729706    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:41.729977 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729940    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:41.731238 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.729501    6033 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.731343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731240    6033 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-workload-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.731343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731260    6033 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-data\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.731343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731254    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:32:41.731549 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731527    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:41.731616 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731593    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:41.731734 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.731668    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7" pod "cf36b503-0a11-463a-b78c-368fabfc6d68" (UID: "cf36b503-0a11-463a-b78c-368fabfc6d68"). InnerVolumeSpecName "kube-api-access-9qmx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:41.832420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832382    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmx7\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-kube-api-access-9qmx7\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.832420 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832409    6033 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-credential-socket\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.832676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832442    6033 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf36b503-0a11-463a-b78c-368fabfc6d68-istiod-ca-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.832676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832452    6033 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-envoy\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.832676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832462    6033 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-podinfo\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.832676 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.832471    6033 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf36b503-0a11-463a-b78c-368fabfc6d68-istio-token\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:41.956987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:41.956958    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 31 00:32:42.034567 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.034523    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\") pod \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") "
Oct 31 00:32:42.034798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.034608    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\" (UniqueName: \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\") pod \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") "
Oct 31 00:32:42.034798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.034647    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\") pod \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\" (UID: \"a9493ebf-b78f-48b3-acac-04cdfcd9b629\") "
Oct 31 00:32:42.036831 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.036802    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6" pod "a9493ebf-b78f-48b3-acac-04cdfcd9b629" (UID: "a9493ebf-b78f-48b3-acac-04cdfcd9b629"). InnerVolumeSpecName "kube-api-access-2c4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:42.036940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.036835    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert" pod "a9493ebf-b78f-48b3-acac-04cdfcd9b629" (UID: "a9493ebf-b78f-48b3-acac-04cdfcd9b629"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:42.036940 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.036908    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs" pod "a9493ebf-b78f-48b3-acac-04cdfcd9b629" (UID: "a9493ebf-b78f-48b3-acac-04cdfcd9b629"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:42.135592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.135545    6033 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-metrics-certs\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:42.135592 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.135587    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4r6\" (UniqueName: \"kubernetes.io/projected/a9493ebf-b78f-48b3-acac-04cdfcd9b629-kube-api-access-2c4r6\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:42.135896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.135604    6033 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9493ebf-b78f-48b3-acac-04cdfcd9b629-webhook-cert\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:32:42.197070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.197014    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:32:42.202697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.202611    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5c3374-openshift-default-d6ff699c7-ppwlm"]
Oct 31 00:32:42.698540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.698508    6033 generic.go:356] "Generic (PLEG): container finished" podID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerID="1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7" exitCode=0
Oct 31 00:32:42.699003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.698560    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c745b59c4-jcxwj"
Oct 31 00:32:42.699003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.698590    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerDied","Data":"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"}
Oct 31 00:32:42.699003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.698644    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c745b59c4-jcxwj" event={"ID":"a9493ebf-b78f-48b3-acac-04cdfcd9b629","Type":"ContainerDied","Data":"678dcd2e78381de7272c44f07feecba23b0dfe247dc4e27b7742d074268d6bc6"}
Oct 31 00:32:42.699003 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.698674    6033 scope.go:122] "RemoveContainer" containerID="1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"
Oct 31 00:32:42.709318 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.709299    6033 scope.go:122] "RemoveContainer" containerID="195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"
Oct 31 00:32:42.717340 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.717315    6033 scope.go:122] "RemoveContainer" containerID="1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"
Oct 31 00:32:42.717652 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:42.717626    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7\": container with ID starting with 1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7 not found: ID does not exist" containerID="1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"
Oct 31 00:32:42.717710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.717666    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7"} err="failed to get container status \"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7\": rpc error: code = NotFound desc = could not find container \"1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7\": container with ID starting with 1d55c6e168e94a81a8448a307fbadca54d7382319e0558a967c06d17ce162da7 not found: ID does not exist"
Oct 31 00:32:42.717710 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.717691    6033 scope.go:122] "RemoveContainer" containerID="195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"
Oct 31 00:32:42.717968 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:42.717948    6033 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9\": container with ID starting with 195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9 not found: ID does not exist" containerID="195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"
Oct 31 00:32:42.718007 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.717975    6033 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9"} err="failed to get container status \"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9\": rpc error: code = NotFound desc = could not find container \"195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9\": container with ID starting with 195430721c74dbae34a94eea75fb0d5b9385d1dc839f7319c0b161bd78c164c9 not found: ID does not exist"
Oct 31 00:32:42.852849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.852809    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 31 00:32:42.857109 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:42.857080    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c745b59c4-jcxwj"]
Oct 31 00:32:43.763927 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:43.763894    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:32:43.828594 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:43.828556    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" path="/var/lib/kubelet/pods/a9493ebf-b78f-48b3-acac-04cdfcd9b629/volumes"
Oct 31 00:32:43.829001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:32:43.828987    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" path="/var/lib/kubelet/pods/cf36b503-0a11-463a-b78c-368fabfc6d68/volumes"
Oct 31 00:32:46.969828 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:32:46.969789    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:33:10.864623 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:33:10.864581    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:33:16.970335 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:33:16.970296    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:33:46.970558 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:33:46.970515    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:34:07.476958 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:07.476915    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:34:16.969270 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:34:16.969219    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:34:28.937296 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:28.937254    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:34:30.265331 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265296    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"]
Oct 31 00:34:30.265748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265727    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.265748 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265739    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265751    6033 cpu_manager.go:414] "containerMap: removing container" podUID="4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" containerName="revoke-keys"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265757    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" containerName="revoke-keys"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265781    6033 cpu_manager.go:414] "containerMap: removing container" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265787    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3489c442-5217-458e-a6ba-84bec2326a7d" containerName="cleanup"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265804    6033 cpu_manager.go:414] "containerMap: removing container" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265810    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265816    6033 cpu_manager.go:414] "containerMap: removing container" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265823    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy"
Oct 31 00:34:30.265827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265830    6033 cpu_manager.go:414] "containerMap: removing container" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265835    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265896    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2ae0c6-5d32-4ac4-ac63-1cbf181fd88b" containerName="main"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265903    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf36b503-0a11-463a-b78c-368fabfc6d68" containerName="istio-proxy"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265911    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bea949e-21bc-4993-8004-9922b5dbd11b" containerName="main"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265919    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265925    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" containerName="revoke-keys"
Oct 31 00:34:30.266145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.265933    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.269137 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.269118    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280212 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280177    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280221    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280312    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280344    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280374 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280365    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280451    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280496    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280550 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280537    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.280651 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.280557    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82wwj\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-kube-api-access-82wwj\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.312321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.312279    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"]
Oct 31 00:34:30.312825 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.312802    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.312926 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.312827    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9493ebf-b78f-48b3-acac-04cdfcd9b629" containerName="manager"
Oct 31 00:34:30.321807 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.321776    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.366583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.366542    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"]
Oct 31 00:34:30.381760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381684    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381830    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381872    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381892    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8ln\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-kube-api-access-ds8ln\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381912    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.381927    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382056    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382118    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382161    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382224    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82wwj\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-kube-api-access-82wwj\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382262    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382283    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382297    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382324    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382412    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.382513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382469    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382530    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382588    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382640    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382671    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.382805    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.383061    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.383505 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.383483    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.384857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.384832    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.385175 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.385155    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.428253 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.428209    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"]
Oct 31 00:34:30.441281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.441244    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.442869 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.442839    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82wwj\" (UniqueName: \"kubernetes.io/projected/26e036ea-2308-4ba1-bd0f-4955e4bc2f52-kube-api-access-82wwj\") pod \"maas-default-gateway-openshift-default-8f6f87f59-znmqk\" (UID: \"26e036ea-2308-4ba1-bd0f-4955e4bc2f52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.483375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483331    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483385    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483491    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483598    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483630    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8ln\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-kube-api-access-ds8ln\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483661    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483716    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483745    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483800    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483841    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-certs\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.483890 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.483879    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-workload-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.484308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.484148    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-data\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.484308 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.484182    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-credential-socket\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.484487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.484470    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.485870 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.485851    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.486196 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.486176    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.543644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.543564    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-istio-token\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.543796 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.543705    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8ln\" (UniqueName: \"kubernetes.io/projected/a4a3244b-0d13-42ad-8857-c3d8b83442f4-kube-api-access-ds8ln\") pod \"maas-default-gateway-openshift-default-8f6f87f59-pqml2\" (UID: \"a4a3244b-0d13-42ad-8857-c3d8b83442f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:30.582803 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.582759    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:30.636283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:30.636248    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:31.016319 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.016283    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"]
Oct 31 00:34:31.051258 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:34:31.051224    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e036ea_2308_4ba1_bd0f_4955e4bc2f52.slice/crio-f95d57f58ab67bd6713135e781d7993391a6009525b1eca21c493dee8b15b698 WatchSource:0}: Error finding container f95d57f58ab67bd6713135e781d7993391a6009525b1eca21c493dee8b15b698: Status 404 returned error can't find the container with id f95d57f58ab67bd6713135e781d7993391a6009525b1eca21c493dee8b15b698
Oct 31 00:34:31.053443 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.053405    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.053548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.053484    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.053548 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.053522    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.079459 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.079423    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"]
Oct 31 00:34:31.083243 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:34:31.083213    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a3244b_0d13_42ad_8857_c3d8b83442f4.slice/crio-794917cd98ce9be8a58087bfb1db61d40838fdb4162c1a3a2dddeb84f85808df WatchSource:0}: Error finding container 794917cd98ce9be8a58087bfb1db61d40838fdb4162c1a3a2dddeb84f85808df: Status 404 returned error can't find the container with id 794917cd98ce9be8a58087bfb1db61d40838fdb4162c1a3a2dddeb84f85808df
Oct 31 00:34:31.086000 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.085955    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.086124 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.086086    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.086166 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.086136    6033 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 00:34:31.233745 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.233709    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" event={"ID":"26e036ea-2308-4ba1-bd0f-4955e4bc2f52","Type":"ContainerStarted","Data":"edaf2f6fc6ba81207ccbebfc58e84ebe418a704dac7be6ac89061a04d00a3d3e"}
Oct 31 00:34:31.233943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.233763    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" event={"ID":"26e036ea-2308-4ba1-bd0f-4955e4bc2f52","Type":"ContainerStarted","Data":"f95d57f58ab67bd6713135e781d7993391a6009525b1eca21c493dee8b15b698"}
Oct 31 00:34:31.235956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.235873    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" event={"ID":"a4a3244b-0d13-42ad-8857-c3d8b83442f4","Type":"ContainerStarted","Data":"b3d83de43cc1aca40d1b817fb7702dd423cdc9e606f474853118317c5272eb03"}
Oct 31 00:34:31.235956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.235926    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" event={"ID":"a4a3244b-0d13-42ad-8857-c3d8b83442f4","Type":"ContainerStarted","Data":"794917cd98ce9be8a58087bfb1db61d40838fdb4162c1a3a2dddeb84f85808df"}
Oct 31 00:34:31.350759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.350634    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podStartSLOduration=1.350619931 podStartE2EDuration="1.350619931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:31.349568997 +0000 UTC m=+3123.696552896" watchObservedRunningTime="2026-10-31 00:34:31.350619931 +0000 UTC m=+3123.697603828"
Oct 31 00:34:31.466832 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.466764    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podStartSLOduration=1.466742747 podStartE2EDuration="1.466742747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:31.465620076 +0000 UTC m=+3123.812603974" watchObservedRunningTime="2026-10-31 00:34:31.466742747 +0000 UTC m=+3123.813726646"
Oct 31 00:34:31.594054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.590128    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:31.637376 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:31.637267    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:32.604058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:32.595068    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:32.604058 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:32.595156    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:32.638402 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:32.638353    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:32.638575 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:32.638415    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:33.585583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:33.585534    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:33.585912 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:33.585877    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:33.637427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:33.637386    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:33.637853 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:33.637453    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:34.586021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:34.585526    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:34.586021 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:34.585596    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:34.639895 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:34.639848    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:34.640346 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:34.639936    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:35.585547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:35.585502    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:35.585907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:35.585866    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:35.641056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:35.638964    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:35.641056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:35.639023    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:36.589056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:36.588732    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:36.589056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:36.588842    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:36.637554 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:36.637506    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:36.637863 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:36.637825    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:37.583425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:37.583382    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:37.584053 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:37.583455    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:37.637511 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:37.637455    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:37.637706 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:37.637552    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:38.584050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:38.583976    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 00:34:38.584472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:38.584082    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 00:34:38.636638 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:38.636597    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Oct 31 00:34:38.636867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:38.636667    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Oct 31 00:34:40.257797 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:40.257753    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:34:40.584415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:40.584307    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:40.584724 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:40.584680    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:40.679230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:40.679179    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:40.679423 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:40.679247    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.626295 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.626240    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-znmqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:41.626867 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.626341    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk" podUID="26e036ea-2308-4ba1-bd0f-4955e4bc2f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.679865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.679823    6033 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8f6f87f59-pqml2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:34:41.680086 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.679901    6033 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2" podUID="a4a3244b-0d13-42ad-8857-c3d8b83442f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:34:41.930784 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.930690    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:41.931201 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.931171    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:41.932271 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:41.932246    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-pqml2"
Oct 31 00:34:42.287197 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:42.287157    6033 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:42.287641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:42.287616    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:42.288563 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:34:42.288544    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8f6f87f59-znmqk"
Oct 31 00:34:46.970470 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:34:46.970432    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:35:00.441605 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:00.441573    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:35:07.046149 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:07.046053    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"]
Oct 31 00:35:07.092629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:07.092593    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0441080d-gwzts"]
Oct 31 00:35:07.828829 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:07.828792    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfc757e-27e7-4dde-ae21-c60549fe3c10" path="/var/lib/kubelet/pods/dcfc757e-27e7-4dde-ae21-c60549fe3c10/volumes"
Oct 31 00:35:16.971452 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:35:16.971410    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:35:17.457066 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:17.457009    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:35:19.050794 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:19.050755    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"]
Oct 31 00:35:19.105050 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:19.104994    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e220b258-k7j4q"]
Oct 31 00:35:19.844423 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:19.844380    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9" path="/var/lib/kubelet/pods/90f0cb5e-d1e4-41a1-a63f-4e9d4ebc7ec9/volumes"
Oct 31 00:35:46.972006 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:35:46.971959    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:35:48.174484 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:48.174450    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:35:58.944480 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:35:58.944440    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:36:11.748362 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:36:11.748325    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:36:16.969724 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:36:16.969685    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:36:30.036407 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:36:30.036137    6033 scope.go:122] "RemoveContainer" containerID="e061a4496444506bbb25c7c7f8eed7a5861cb3d6d311ea0c081b068a1a5d670f"
Oct 31 00:36:30.051341 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:36:30.051314    6033 scope.go:122] "RemoveContainer" containerID="358db2022674d77299a57ac1fc75f28f503be1a058f5a40c0e55bf64381bb252"
Oct 31 00:36:36.648670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:36:36.648573    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:36:46.459110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:36:46.459067    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:36:46.969064 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:36:46.968997    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:37:04.845793 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:04.845751    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:37:15.056774 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:15.056733    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"]
Oct 31 00:37:15.103462 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:15.103425    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3073a8d8-sgh5h"]
Oct 31 00:37:15.828192 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:15.828154    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d03c369-01a5-4a1f-88a9-48c2d5dd27fd" path="/var/lib/kubelet/pods/4d03c369-01a5-4a1f-88a9-48c2d5dd27fd/volumes"
Oct 31 00:37:16.969924 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:37:16.969881    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:37:17.475732 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:17.475693    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:37:38.576857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:37:38.576814    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:37:46.970131 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:37:46.970088    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:38:00.561048 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:00.560996    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:38:13.574429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:13.574341    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:38:16.970118 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:38:16.970076    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:38:24.276800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:24.276756    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:38:30.090517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:30.084994    6033 scope.go:122] "RemoveContainer" containerID="2a8e68178f29aefca2bde940216867e73d9aaf2575c618cc0503e210d8cbef5c"
Oct 31 00:38:46.971296 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:38:46.971250    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:38:47.637658 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:47.637614    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:38:57.973993 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:38:57.973951    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:39:14.632398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:39:14.632354    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:39:16.993869 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:39:16.993825    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:39:36.055933 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:39:36.055832    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:39:46.734521 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:39:46.734476    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:39:46.970473 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:39:46.970434    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:40:05.523100 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:40:05.523055    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:40:16.327560 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:40:16.327518    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:40:16.970930 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:40:16.970893    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:40:42.352424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:40:42.352348    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:40:46.970912 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:40:46.970868    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:41:01.046322 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:01.046277    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:41:16.969469 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:41:16.969423    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:41:26.446330 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:26.446283    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:41:40.626017 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.625976    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"]
Oct 31 00:41:40.629603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.629578    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.672522 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.672483    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.672717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.672548    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.672717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.672626    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gs9\" (UniqueName: \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.678644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.678599    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:41:40.725184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.725143    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:41:40.725184 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.725143    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:41:40.725418 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.725212    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:41:40.726629 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.726604    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"]
Oct 31 00:41:40.773541 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.773491    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.773761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.773555    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.773761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.773598    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gs9\" (UniqueName: \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.774281 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.774258    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.776191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.776164    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.831659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.831618    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gs9\" (UniqueName: \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\") pod \"maas-api-revoke-sub-0ab8abd9-k4pdz\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:40.943876 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:40.943785    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:41.314460 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:41.314423    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"]
Oct 31 00:41:41.317894 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:41:41.317853    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03235b8a_80d5_4362_a296_75f5969ac22f.slice/crio-199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8 WatchSource:0}: Error finding container 199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8: Status 404 returned error can't find the container with id 199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8
Oct 31 00:41:41.507545 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:41.507504    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz" event={"ID":"03235b8a-80d5-4362-a296-75f5969ac22f","Type":"ContainerStarted","Data":"199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8"}
Oct 31 00:41:42.513971 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:42.513937    6033 generic.go:356] "Generic (PLEG): container finished" podID="03235b8a-80d5-4362-a296-75f5969ac22f" containerID="32723ff4544a7d35a1bd39e8e01b28536a625c251dea2aae01a5b1f7673825e2" exitCode=0
Oct 31 00:41:42.514422 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:42.514043    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz" event={"ID":"03235b8a-80d5-4362-a296-75f5969ac22f","Type":"ContainerDied","Data":"32723ff4544a7d35a1bd39e8e01b28536a625c251dea2aae01a5b1f7673825e2"}
Oct 31 00:41:43.660921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.660888    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:43.702997 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.702954    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\") pod \"03235b8a-80d5-4362-a296-75f5969ac22f\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") "
Oct 31 00:41:43.703222 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.703101    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\") pod \"03235b8a-80d5-4362-a296-75f5969ac22f\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") "
Oct 31 00:41:43.703222 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.703206    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\" (UniqueName: \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\") pod \"03235b8a-80d5-4362-a296-75f5969ac22f\" (UID: \"03235b8a-80d5-4362-a296-75f5969ac22f\") "
Oct 31 00:41:43.703439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.703415    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca" pod "03235b8a-80d5-4362-a296-75f5969ac22f" (UID: "03235b8a-80d5-4362-a296-75f5969ac22f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:43.703558 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.703540    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:43.705230 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.705204    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca" pod "03235b8a-80d5-4362-a296-75f5969ac22f" (UID: "03235b8a-80d5-4362-a296-75f5969ac22f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:41:43.705309 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.705261    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9" pod "03235b8a-80d5-4362-a296-75f5969ac22f" (UID: "03235b8a-80d5-4362-a296-75f5969ac22f"). InnerVolumeSpecName "kube-api-access-p5gs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:43.804708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.804604    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03235b8a-80d5-4362-a296-75f5969ac22f-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:43.804708 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:43.804657    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gs9\" (UniqueName: \"kubernetes.io/projected/03235b8a-80d5-4362-a296-75f5969ac22f-kube-api-access-p5gs9\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:44.524987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:44.524942    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"
Oct 31 00:41:44.524987 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:44.524972    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz" event={"ID":"03235b8a-80d5-4362-a296-75f5969ac22f","Type":"ContainerDied","Data":"199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8"}
Oct 31 00:41:44.525248 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:44.525042    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199b3d19ad99f72d22b4746800ec459255045442dcdf84ccd015e6b55d1dffc8"
Oct 31 00:41:46.969408 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:41:46.969371    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:41:52.918085 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.918049    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"]
Oct 31 00:41:52.918513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.918483    6033 cpu_manager.go:414] "containerMap: removing container" podUID="03235b8a-80d5-4362-a296-75f5969ac22f" containerName="revoke-keys"
Oct 31 00:41:52.918513 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.918494    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03235b8a-80d5-4362-a296-75f5969ac22f" containerName="revoke-keys"
Oct 31 00:41:52.918586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.918567    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="03235b8a-80d5-4362-a296-75f5969ac22f" containerName="revoke-keys"
Oct 31 00:41:52.921839 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.921806    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:52.970880 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:52.970846    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:41:53.002446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.002399    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.002634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.002464    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqfm\" (UniqueName: \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.002634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.002560    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.017283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.017247    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:41:53.017283 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.017258    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:41:53.017529 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.017315    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:41:53.018845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.018817    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"]
Oct 31 00:41:53.103428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.103384    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.103659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.103456    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqfm\" (UniqueName: \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.103659 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.103562    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.104191 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.104167    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.105917 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.105887    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.161552 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.161514    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqfm\" (UniqueName: \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.244858 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.244774    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:53.612809 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:53.612762    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"]
Oct 31 00:41:53.616137 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:41:53.616105    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7135431_0e9d_4c5e_9788_92cd14df2a33.slice/crio-89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115 WatchSource:0}: Error finding container 89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115: Status 404 returned error can't find the container with id 89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115
Oct 31 00:41:54.576099 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:54.576067    6033 generic.go:356] "Generic (PLEG): container finished" podID="a7135431-0e9d-4c5e-9788-92cd14df2a33" containerID="938ef0da1a6a7b668c0097d659623c9af45144f3cb6083c7776aaa22e080b65e" exitCode=0
Oct 31 00:41:54.576510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:54.576149    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z" event={"ID":"a7135431-0e9d-4c5e-9788-92cd14df2a33","Type":"ContainerDied","Data":"938ef0da1a6a7b668c0097d659623c9af45144f3cb6083c7776aaa22e080b65e"}
Oct 31 00:41:54.576510 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:54.576189    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z" event={"ID":"a7135431-0e9d-4c5e-9788-92cd14df2a33","Type":"ContainerStarted","Data":"89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115"}
Oct 31 00:41:55.656458 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.656416    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:41:55.724369 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.724333    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:41:55.832678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.832597    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\" (UniqueName: \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\") pod \"a7135431-0e9d-4c5e-9788-92cd14df2a33\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") "
Oct 31 00:41:55.832678 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.832651    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\") pod \"a7135431-0e9d-4c5e-9788-92cd14df2a33\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") "
Oct 31 00:41:55.832902 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.832693    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\") pod \"a7135431-0e9d-4c5e-9788-92cd14df2a33\" (UID: \"a7135431-0e9d-4c5e-9788-92cd14df2a33\") "
Oct 31 00:41:55.833269 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.833240    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca" pod "a7135431-0e9d-4c5e-9788-92cd14df2a33" (UID: "a7135431-0e9d-4c5e-9788-92cd14df2a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:55.833908 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.833878    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:55.835272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.835243    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca" pod "a7135431-0e9d-4c5e-9788-92cd14df2a33" (UID: "a7135431-0e9d-4c5e-9788-92cd14df2a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:41:55.835388 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.835274    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm" pod "a7135431-0e9d-4c5e-9788-92cd14df2a33" (UID: "a7135431-0e9d-4c5e-9788-92cd14df2a33"). InnerVolumeSpecName "kube-api-access-jbqfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:55.935116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.935068    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqfm\" (UniqueName: \"kubernetes.io/projected/a7135431-0e9d-4c5e-9788-92cd14df2a33-kube-api-access-jbqfm\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:55.935116 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:55.935115    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7135431-0e9d-4c5e-9788-92cd14df2a33-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:41:56.586546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:56.586508    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z" event={"ID":"a7135431-0e9d-4c5e-9788-92cd14df2a33","Type":"ContainerDied","Data":"89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115"}
Oct 31 00:41:56.586728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:56.586560    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89782a24b45049f9851de45867f4cbd183538a59897b7b39a6daf221c3016115"
Oct 31 00:41:56.586728 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:41:56.586524    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"
Oct 31 00:42:09.319272 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:09.319228    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"]
Oct 31 00:42:09.365845 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:09.365808    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0e6572-gtb4z"]
Oct 31 00:42:09.827671 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:09.827636    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7135431-0e9d-4c5e-9788-92cd14df2a33" path="/var/lib/kubelet/pods/a7135431-0e9d-4c5e-9788-92cd14df2a33/volumes"
Oct 31 00:42:11.834477 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:11.834440    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:42:16.971276 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:42:16.971205    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:42:32.258023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:32.257984    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:42:43.036067 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:42:43.035967    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:42:46.969256 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:42:46.969219    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:43:16.970793 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:43:16.970736    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:43:46.968607 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:43:46.968565    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:44:16.160723 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:44:16.160632    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:44:16.971480 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:44:16.971420    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:44:33.667518 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:44:33.667482    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:44:43.933177 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:44:43.933135    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:44:46.969236 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:44:46.969195    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:45:02.576240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:45:02.576203    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:45:16.969821 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:45:16.969783    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:45:21.758110 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:45:21.758071    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:45:39.958717 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:45:39.958608    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:45:46.970230 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:45:46.970185    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:46:05.932176 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:05.932135    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:46:16.970313 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:46:16.970266    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:46:21.008799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.008760    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"]
Oct 31 00:46:21.009240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.009221    6033 cpu_manager.go:414] "containerMap: removing container" podUID="a7135431-0e9d-4c5e-9788-92cd14df2a33" containerName="revoke-keys"
Oct 31 00:46:21.009240 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.009233    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7135431-0e9d-4c5e-9788-92cd14df2a33" containerName="revoke-keys"
Oct 31 00:46:21.009321 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.009303    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7135431-0e9d-4c5e-9788-92cd14df2a33" containerName="revoke-keys"
Oct 31 00:46:21.012652 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.012629    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.061887 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.061844    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:46:21.108425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.108380    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:46:21.108425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.108380    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:46:21.108425 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.108380    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:46:21.109995 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.109964    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"]
Oct 31 00:46:21.167757 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.167712    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.167949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.167775    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.167949 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.167866    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84ff\" (UniqueName: \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.269424 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.269328    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.269627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.269499    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.269627 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.269559    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84ff\" (UniqueName: \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.270212 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.270191    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.271850 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.271825    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.327776 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.327744    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84ff\" (UniqueName: \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\") pod \"maas-api-revoke-sub-c06b5bd0-h4g2h\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.626257 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.626166    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:21.993506 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:21.993473    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"]
Oct 31 00:46:21.996166 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:46:21.996134    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d8a244_5e8e_4ba1_9129_4e985f481486.slice/crio-6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c WatchSource:0}: Error finding container 6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c: Status 404 returned error can't find the container with id 6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c
Oct 31 00:46:22.950539 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:22.950498    6033 generic.go:356] "Generic (PLEG): container finished" podID="61d8a244-5e8e-4ba1-9129-4e985f481486" containerID="ea256716a9d192201e33dfc552bc89906e2b726ca5c21d43bceea4382566cb66" exitCode=0
Oct 31 00:46:22.950960 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:22.950588    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h" event={"ID":"61d8a244-5e8e-4ba1-9129-4e985f481486","Type":"ContainerDied","Data":"ea256716a9d192201e33dfc552bc89906e2b726ca5c21d43bceea4382566cb66"}
Oct 31 00:46:22.950960 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:22.950643    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h" event={"ID":"61d8a244-5e8e-4ba1-9129-4e985f481486","Type":"ContainerStarted","Data":"6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c"}
Oct 31 00:46:24.099749 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.099719    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:24.198547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.198512    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\" (UniqueName: \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\") pod \"61d8a244-5e8e-4ba1-9129-4e985f481486\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") "
Oct 31 00:46:24.198547 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.198552    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\") pod \"61d8a244-5e8e-4ba1-9129-4e985f481486\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") "
Oct 31 00:46:24.198773 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.198635    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\") pod \"61d8a244-5e8e-4ba1-9129-4e985f481486\" (UID: \"61d8a244-5e8e-4ba1-9129-4e985f481486\") "
Oct 31 00:46:24.198955 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.198931    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca" pod "61d8a244-5e8e-4ba1-9129-4e985f481486" (UID: "61d8a244-5e8e-4ba1-9129-4e985f481486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:24.200765 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.200738    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff" pod "61d8a244-5e8e-4ba1-9129-4e985f481486" (UID: "61d8a244-5e8e-4ba1-9129-4e985f481486"). InnerVolumeSpecName "kube-api-access-l84ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:24.200897 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.200784    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca" pod "61d8a244-5e8e-4ba1-9129-4e985f481486" (UID: "61d8a244-5e8e-4ba1-9129-4e985f481486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:24.299966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.299925    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84ff\" (UniqueName: \"kubernetes.io/projected/61d8a244-5e8e-4ba1-9129-4e985f481486-kube-api-access-l84ff\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:24.299966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.299959    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:24.299966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.299971    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d8a244-5e8e-4ba1-9129-4e985f481486-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:24.962735 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.962693    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h" event={"ID":"61d8a244-5e8e-4ba1-9129-4e985f481486","Type":"ContainerDied","Data":"6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c"}
Oct 31 00:46:24.962735 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.962717    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"
Oct 31 00:46:24.962963 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:24.962788    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6467784c48d28dcf352436ed989c93a13806cb2581f21ab038e69149b9cdf65c"
Oct 31 00:46:24.993686 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:46:24.993646    6033 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d8a244_5e8e_4ba1_9129_4e985f481486.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:46:32.732314 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.732269    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"]
Oct 31 00:46:32.732812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.732706    6033 cpu_manager.go:414] "containerMap: removing container" podUID="61d8a244-5e8e-4ba1-9129-4e985f481486" containerName="revoke-keys"
Oct 31 00:46:32.732812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.732718    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d8a244-5e8e-4ba1-9129-4e985f481486" containerName="revoke-keys"
Oct 31 00:46:32.732812 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.732786    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d8a244-5e8e-4ba1-9129-4e985f481486" containerName="revoke-keys"
Oct 31 00:46:32.735806 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.735785    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.776256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.776210    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.776469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.776362    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r7h\" (UniqueName: \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.776469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.776397    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.784961 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.784928    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:46:32.831561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.831518    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:46:32.831561 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.831561    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:46:32.831851 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.831626    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fdzs\""
Oct 31 00:46:32.833264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.833234    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"]
Oct 31 00:46:32.877683 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.877623    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.877866 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.877847    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.878078 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.878022    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r7h\" (UniqueName: \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.878508 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.878478    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.880449 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.880427    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:32.936465 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:32.936433    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r7h\" (UniqueName: \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:33.049056 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:33.048995    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:33.420405 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:33.420132    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"]
Oct 31 00:46:33.423546 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:46:33.423507    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f63a65d_0e12_40f2_93f0_c20def511251.slice/crio-f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711 WatchSource:0}: Error finding container f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711: Status 404 returned error can't find the container with id f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711
Oct 31 00:46:34.006609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:34.006567    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8" event={"ID":"1f63a65d-0e12-40f2-93f0-c20def511251","Type":"ContainerStarted","Data":"d27446f2c644024386383557a6bfe523d9fd374432d4defeab372c9314ad734a"}
Oct 31 00:46:34.007199 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:34.006638    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8" event={"ID":"1f63a65d-0e12-40f2-93f0-c20def511251","Type":"ContainerStarted","Data":"f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711"}
Oct 31 00:46:34.024243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:34.024196    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:46:34.130487 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:34.130420    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8" podStartSLOduration=2.1304056510000002 podStartE2EDuration="2.130405651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:34.12887884 +0000 UTC m=+3846.475862739" watchObservedRunningTime="2026-10-31 00:46:34.130405651 +0000 UTC m=+3846.477389548"
Oct 31 00:46:35.013546 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:35.013509    6033 generic.go:356] "Generic (PLEG): container finished" podID="1f63a65d-0e12-40f2-93f0-c20def511251" containerID="d27446f2c644024386383557a6bfe523d9fd374432d4defeab372c9314ad734a" exitCode=0
Oct 31 00:46:35.013956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:35.013594    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8" event={"ID":"1f63a65d-0e12-40f2-93f0-c20def511251","Type":"ContainerDied","Data":"d27446f2c644024386383557a6bfe523d9fd374432d4defeab372c9314ad734a"}
Oct 31 00:46:36.153777 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.153749    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:36.209822 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.209780    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\") pod \"1f63a65d-0e12-40f2-93f0-c20def511251\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") "
Oct 31 00:46:36.210023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.209829    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\" (UniqueName: \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\") pod \"1f63a65d-0e12-40f2-93f0-c20def511251\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") "
Oct 31 00:46:36.210023 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.209892    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\") pod \"1f63a65d-0e12-40f2-93f0-c20def511251\" (UID: \"1f63a65d-0e12-40f2-93f0-c20def511251\") "
Oct 31 00:46:36.210216 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.210195    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca" pod "1f63a65d-0e12-40f2-93f0-c20def511251" (UID: "1f63a65d-0e12-40f2-93f0-c20def511251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:36.212063 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.212039    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca" pod "1f63a65d-0e12-40f2-93f0-c20def511251" (UID: "1f63a65d-0e12-40f2-93f0-c20def511251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:46:36.212160 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.212119    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h" pod "1f63a65d-0e12-40f2-93f0-c20def511251" (UID: "1f63a65d-0e12-40f2-93f0-c20def511251"). InnerVolumeSpecName "kube-api-access-h8r7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:36.311660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.311557    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:36.311660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.311595    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r7h\" (UniqueName: \"kubernetes.io/projected/1f63a65d-0e12-40f2-93f0-c20def511251-kube-api-access-h8r7h\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:36.311660 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:36.311610    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f63a65d-0e12-40f2-93f0-c20def511251-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:46:37.024805 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:37.024771    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8" event={"ID":"1f63a65d-0e12-40f2-93f0-c20def511251","Type":"ContainerDied","Data":"f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711"}
Oct 31 00:46:37.024967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:37.024809    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"
Oct 31 00:46:37.024967 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:37.024821    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f65ca2764c791466331769884e5c8d517da0538f894367ab638f1b81a2e08711"
Oct 31 00:46:44.047583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:44.047546    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"]
Oct 31 00:46:44.093983 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:44.093943    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab8abd9-k4pdz"]
Oct 31 00:46:45.827808 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:45.827770    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03235b8a-80d5-4362-a296-75f5969ac22f" path="/var/lib/kubelet/pods/03235b8a-80d5-4362-a296-75f5969ac22f/volumes"
Oct 31 00:46:46.969560 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:46:46.969515    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:46:48.935247 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:48.935206    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"]
Oct 31 00:46:48.937835 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:48.937805    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e98014-t97p8"]
Oct 31 00:46:49.144980 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:49.144943    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:46:49.828427 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:46:49.828388    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f63a65d-0e12-40f2-93f0-c20def511251" path="/var/lib/kubelet/pods/1f63a65d-0e12-40f2-93f0-c20def511251/volumes"
Oct 31 00:47:08.078625 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:47:08.078534    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:47:16.969822 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:47:16.969780    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:47:46.971493 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:47:46.971451    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:47:52.570962 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:47:52.570913    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:48:16.969786 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:48:16.969742    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:48:20.761882 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:20.761831    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:48:30.187494 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:30.187456    6033 scope.go:122] "RemoveContainer" containerID="938ef0da1a6a7b668c0097d659623c9af45144f3cb6083c7776aaa22e080b65e"
Oct 31 00:48:30.196613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:30.196589    6033 scope.go:122] "RemoveContainer" containerID="32723ff4544a7d35a1bd39e8e01b28536a625c251dea2aae01a5b1f7673825e2"
Oct 31 00:48:36.241497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:36.241392    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:48:46.347762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:46.347719    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:48:46.969406 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:48:46.969368    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:48:54.570695 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:48:54.570647    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:49:05.262155 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:49:05.262113    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:49:16.970576 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:49:16.970534    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:49:27.070855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:49:27.070814    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:49:46.969087 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:49:46.969018    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:49:51.956904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:49:51.956867    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:50:15.233353 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.233255    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"]
Oct 31 00:50:15.234054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.234017    6033 cpu_manager.go:414] "containerMap: removing container" podUID="1f63a65d-0e12-40f2-93f0-c20def511251" containerName="revoke-keys"
Oct 31 00:50:15.234135 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.234057    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f63a65d-0e12-40f2-93f0-c20def511251" containerName="revoke-keys"
Oct 31 00:50:15.234190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.234154    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f63a65d-0e12-40f2-93f0-c20def511251" containerName="revoke-keys"
Oct 31 00:50:15.237884 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.237855    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.335414 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.335366    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:50:15.335614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.335429    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:50:15.335614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.335447    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkrrm\""
Oct 31 00:50:15.335614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.335503    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:50:15.336849 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.336826    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"]
Oct 31 00:50:15.383106 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.383061    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.383287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.383171    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.383343 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.383329    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcl7t\" (UniqueName: \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.484234 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.484132    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.484234 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.484184    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.484450 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.484274    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcl7t\" (UniqueName: \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.485089 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.485057    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.486635 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.486613    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.541956 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.541924    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcl7t\" (UniqueName: \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\") pod \"maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.551752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.551713    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:15.920698 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:15.920645    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"]
Oct 31 00:50:15.923858 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:50:15.923823    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7f05b4_439b_40af_ab1b_b1ba2ab92af4.slice/crio-90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89 WatchSource:0}: Error finding container 90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89: Status 404 returned error can't find the container with id 90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89
Oct 31 00:50:16.251543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:16.251508    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5" event={"ID":"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4","Type":"ContainerStarted","Data":"9407304e8287dd7f25e149cf916781d004c7a33ab2f47799583759a0ccebbcfd"}
Oct 31 00:50:16.251954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:16.251567    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5" event={"ID":"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4","Type":"ContainerStarted","Data":"90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89"}
Oct 31 00:50:16.361208 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:16.361151    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5" podStartSLOduration=1.3611341989999999 podStartE2EDuration="1.361134199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:16.360153583 +0000 UTC m=+4068.707137480" watchObservedRunningTime="2026-10-31 00:50:16.361134199 +0000 UTC m=+4068.708118098"
Oct 31 00:50:16.970245 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:50:16.970205    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:50:17.258361 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:17.258326    6033 generic.go:356] "Generic (PLEG): container finished" podID="ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" containerID="9407304e8287dd7f25e149cf916781d004c7a33ab2f47799583759a0ccebbcfd" exitCode=0
Oct 31 00:50:17.258798 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:17.258404    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5" event={"ID":"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4","Type":"ContainerDied","Data":"9407304e8287dd7f25e149cf916781d004c7a33ab2f47799583759a0ccebbcfd"}
Oct 31 00:50:18.403046 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.403007    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:18.514071 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.513942    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\") pod \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") "
Oct 31 00:50:18.514266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.514092    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\" (UniqueName: \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\") pod \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") "
Oct 31 00:50:18.514266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.514192    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\") pod \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\" (UID: \"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4\") "
Oct 31 00:50:18.514762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.514705    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca" pod "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" (UID: "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:50:18.516758 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.516722    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t" pod "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" (UID: "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4"). InnerVolumeSpecName "kube-api-access-dcl7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:18.516898 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.516835    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca" pod "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" (UID: "ca7f05b4-439b-40af-ab1b-b1ba2ab92af4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:18.615788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.615737    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcl7t\" (UniqueName: \"kubernetes.io/projected/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-kube-api-access-dcl7t\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:18.615788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.615774    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:18.615788 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:18.615784    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:19.269726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:19.269698    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"
Oct 31 00:50:19.269907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:19.269697    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5" event={"ID":"ca7f05b4-439b-40af-ab1b-b1ba2ab92af4","Type":"ContainerDied","Data":"90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89"}
Oct 31 00:50:19.269907 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:19.269829    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b3ba192356ab10814e44ecdcfa2e664ae433b8addbc4fe5bc405b8bac94c89"
Oct 31 00:50:36.745609 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:36.745566    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"]
Oct 31 00:50:36.792054 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:36.792002    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8214dff9-wd5v5"]
Oct 31 00:50:37.828170 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:37.828140    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" path="/var/lib/kubelet/pods/ca7f05b4-439b-40af-ab1b-b1ba2ab92af4/volumes"
Oct 31 00:50:38.880688 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:50:38.880650    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:50:46.969336 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:50:46.969296    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:51:01.938439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:51:01.938396    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:51:16.970320 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:51:16.970279    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:51:25.042662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:51:25.042619    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"]
Oct 31 00:51:25.088996 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:51:25.088959    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06b5bd0-h4g2h"]
Oct 31 00:51:25.827923 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:51:25.827887    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d8a244-5e8e-4ba1-9129-4e985f481486" path="/var/lib/kubelet/pods/61d8a244-5e8e-4ba1-9129-4e985f481486/volumes"
Oct 31 00:51:46.968629 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:51:46.968543    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:52:16.970137 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:52:16.970096    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:52:30.244681 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:30.244644    6033 scope.go:122] "RemoveContainer" containerID="ea256716a9d192201e33dfc552bc89906e2b726ca5c21d43bceea4382566cb66"
Oct 31 00:52:32.507018 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.506982    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"]
Oct 31 00:52:32.507442 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.507398    6033 cpu_manager.go:414] "containerMap: removing container" podUID="ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" containerName="revoke-keys"
Oct 31 00:52:32.507442 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.507410    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" containerName="revoke-keys"
Oct 31 00:52:32.507524 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.507513    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca7f05b4-439b-40af-ab1b-b1ba2ab92af4" containerName="revoke-keys"
Oct 31 00:52:32.512294 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.512269    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.557531 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.557485    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.557752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.557549    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.557752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.557686    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6lf\" (UniqueName: \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.565984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.565948    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"]
Oct 31 00:52:32.610811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.610746    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:52:32.610811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.610767    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:52:32.610811 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.610775    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkrrm\""
Oct 31 00:52:32.611079 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.610873    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:52:32.658813 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.658770    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.659077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.658829    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.659077 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.658856    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6lf\" (UniqueName: \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.659588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.659562    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.661485 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.661462    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.717256 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.717217    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6lf\" (UniqueName: \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:32.825486 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:32.825401    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:33.194697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:33.194667    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"]
Oct 31 00:52:33.197310 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:52:33.197280    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47749b39_7bdb_4986_a07a_4ec77b05dc2b.slice/crio-142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587 WatchSource:0}: Error finding container 142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587: Status 404 returned error can't find the container with id 142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587
Oct 31 00:52:33.948138 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:33.948097    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56" event={"ID":"47749b39-7bdb-4986-a07a-4ec77b05dc2b","Type":"ContainerStarted","Data":"fd8bf4a424003ebcbc71accf1b1e90bc4f9e1b6b3fbe418cc7ceb07feef0edcc"}
Oct 31 00:52:33.948559 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:33.948158    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56" event={"ID":"47749b39-7bdb-4986-a07a-4ec77b05dc2b","Type":"ContainerStarted","Data":"142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587"}
Oct 31 00:52:34.057588 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:34.057540    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56" podStartSLOduration=2.057524798 podStartE2EDuration="2.057524798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:34.056953255 +0000 UTC m=+4206.403937152" watchObservedRunningTime="2026-10-31 00:52:34.057524798 +0000 UTC m=+4206.404508697"
Oct 31 00:52:34.956540 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:34.956505    6033 generic.go:356] "Generic (PLEG): container finished" podID="47749b39-7bdb-4986-a07a-4ec77b05dc2b" containerID="fd8bf4a424003ebcbc71accf1b1e90bc4f9e1b6b3fbe418cc7ceb07feef0edcc" exitCode=0
Oct 31 00:52:34.956954 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:34.956593    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56" event={"ID":"47749b39-7bdb-4986-a07a-4ec77b05dc2b","Type":"ContainerDied","Data":"fd8bf4a424003ebcbc71accf1b1e90bc4f9e1b6b3fbe418cc7ceb07feef0edcc"}
Oct 31 00:52:36.107877 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.107842    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:36.199697 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.199651    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\") pod \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") "
Oct 31 00:52:36.199896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.199735    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\") pod \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") "
Oct 31 00:52:36.199896 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.199794    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\" (UniqueName: \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\") pod \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\" (UID: \"47749b39-7bdb-4986-a07a-4ec77b05dc2b\") "
Oct 31 00:52:36.200211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.200180    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca" pod "47749b39-7bdb-4986-a07a-4ec77b05dc2b" (UID: "47749b39-7bdb-4986-a07a-4ec77b05dc2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:52:36.202139 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.202097    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca" pod "47749b39-7bdb-4986-a07a-4ec77b05dc2b" (UID: "47749b39-7bdb-4986-a07a-4ec77b05dc2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:36.202266 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.202183    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf" pod "47749b39-7bdb-4986-a07a-4ec77b05dc2b" (UID: "47749b39-7bdb-4986-a07a-4ec77b05dc2b"). InnerVolumeSpecName "kube-api-access-tp6lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:36.301497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.301454    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:36.301497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.301490    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47749b39-7bdb-4986-a07a-4ec77b05dc2b-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:36.301497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.301501    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6lf\" (UniqueName: \"kubernetes.io/projected/47749b39-7bdb-4986-a07a-4ec77b05dc2b-kube-api-access-tp6lf\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:36.968779 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.968741    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56" event={"ID":"47749b39-7bdb-4986-a07a-4ec77b05dc2b","Type":"ContainerDied","Data":"142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587"}
Oct 31 00:52:36.969002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.968806    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="142ab1f3567e805f030f8bd5224a62272c63bdea15a75acd5c42920228de2587"
Oct 31 00:52:36.969002 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:36.968756    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"
Oct 31 00:52:46.969978 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:52:46.969941    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:52:57.287810 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:57.287702    6033 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"]
Oct 31 00:52:57.290415 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:57.290382    6033 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a3c14c-67t56"]
Oct 31 00:52:57.828297 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:57.828263    6033 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47749b39-7bdb-4986-a07a-4ec77b05dc2b" path="/var/lib/kubelet/pods/47749b39-7bdb-4986-a07a-4ec77b05dc2b/volumes"
Oct 31 00:52:59.482131 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:52:59.482089    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:53:12.745674 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:12.745587    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:53:16.969652 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:53:16.969611    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:53:23.381569 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:23.381530    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:53:30.265461 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:30.265423    6033 scope.go:122] "RemoveContainer" containerID="d27446f2c644024386383557a6bfe523d9fd374432d4defeab372c9314ad734a"
Oct 31 00:53:31.599130 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.599093    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"]
Oct 31 00:53:31.599626 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.599607    6033 cpu_manager.go:414] "containerMap: removing container" podUID="47749b39-7bdb-4986-a07a-4ec77b05dc2b" containerName="revoke-keys"
Oct 31 00:53:31.599672 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.599628    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47749b39-7bdb-4986-a07a-4ec77b05dc2b" containerName="revoke-keys"
Oct 31 00:53:31.599782 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.599768    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="47749b39-7bdb-4986-a07a-4ec77b05dc2b" containerName="revoke-keys"
Oct 31 00:53:31.604722 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.604697    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.657603 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.657567    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:53:31.703498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.703457    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:53:31.703498 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.703489    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:53:31.703761 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.703539    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkrrm\""
Oct 31 00:53:31.704848 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.704828    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"]
Oct 31 00:53:31.720512 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.720480    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.720667 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.720534    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jt2z\" (UniqueName: \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.720726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.720679    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.821423 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.821388    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jt2z\" (UniqueName: \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.821619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.821464    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.821619 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.821508    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.822287 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.822260    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.824076 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.824050    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.887264 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.887189    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jt2z\" (UniqueName: \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\") pod \"maas-api-revoke-sub-f03629db-mcncs\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:31.917517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:31.917486    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:32.287277 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:32.287097    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"]
Oct 31 00:53:32.290567 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:53:32.290530    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5c5ef7_644d_4e3b_9cc3_050399c14768.slice/crio-e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d WatchSource:0}: Error finding container e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d: Status 404 returned error can't find the container with id e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d
Oct 31 00:53:33.288614 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:33.288577    6033 generic.go:356] "Generic (PLEG): container finished" podID="5b5c5ef7-644d-4e3b-9cc3-050399c14768" containerID="3e3c08403d2f5f9c450b28993cd179aebd673941a086c5410eca1b6d1d817b18" exitCode=0
Oct 31 00:53:33.289055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:33.288658    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs" event={"ID":"5b5c5ef7-644d-4e3b-9cc3-050399c14768","Type":"ContainerDied","Data":"3e3c08403d2f5f9c450b28993cd179aebd673941a086c5410eca1b6d1d817b18"}
Oct 31 00:53:33.289055 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:33.288709    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs" event={"ID":"5b5c5ef7-644d-4e3b-9cc3-050399c14768","Type":"ContainerStarted","Data":"e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d"}
Oct 31 00:53:34.437454 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.437423    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:34.549263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.549177    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\") pod \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") "
Oct 31 00:53:34.549263 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.549245    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\" (UniqueName: \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\") pod \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") "
Oct 31 00:53:34.549475 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.549304    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\") pod \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\" (UID: \"5b5c5ef7-644d-4e3b-9cc3-050399c14768\") "
Oct 31 00:53:34.549726 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.549704    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca" pod "5b5c5ef7-644d-4e3b-9cc3-050399c14768" (UID: "5b5c5ef7-644d-4e3b-9cc3-050399c14768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:34.551457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.551430    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z" pod "5b5c5ef7-644d-4e3b-9cc3-050399c14768" (UID: "5b5c5ef7-644d-4e3b-9cc3-050399c14768"). InnerVolumeSpecName "kube-api-access-9jt2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:34.551457 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.551433    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca" pod "5b5c5ef7-644d-4e3b-9cc3-050399c14768" (UID: "5b5c5ef7-644d-4e3b-9cc3-050399c14768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:34.651186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.651141    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:34.651186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.651184    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b5c5ef7-644d-4e3b-9cc3-050399c14768-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:34.651186 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:34.651195    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jt2z\" (UniqueName: \"kubernetes.io/projected/5b5c5ef7-644d-4e3b-9cc3-050399c14768-kube-api-access-9jt2z\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:35.300014 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:35.299987    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs"
Oct 31 00:53:35.300190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:35.299986    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-mcncs" event={"ID":"5b5c5ef7-644d-4e3b-9cc3-050399c14768","Type":"ContainerDied","Data":"e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d"}
Oct 31 00:53:35.300190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:35.300113    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e827fe3b7278170a01aa6196d913245da028b059e43feac9a26a683fe040e58d"
Oct 31 00:53:44.358220 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:44.358183    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:53:46.969791 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:53:46.969752    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:53:52.638970 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.638928    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"]
Oct 31 00:53:52.639566 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.639543    6033 cpu_manager.go:414] "containerMap: removing container" podUID="5b5c5ef7-644d-4e3b-9cc3-050399c14768" containerName="revoke-keys"
Oct 31 00:53:52.639655 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.639568    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5c5ef7-644d-4e3b-9cc3-050399c14768" containerName="revoke-keys"
Oct 31 00:53:52.639732 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.639720    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5c5ef7-644d-4e3b-9cc3-050399c14768" containerName="revoke-keys"
Oct 31 00:53:52.643145 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.643119    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.695070 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.695014    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:53:52.740724 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.740681    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:53:52.740724 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.740681    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkrrm\""
Oct 31 00:53:52.740964 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.740757    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:53:52.742246 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.742220    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"]
Oct 31 00:53:52.758220 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.758181    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586sj\" (UniqueName: \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.758401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.758319    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.758401 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.758391    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.859429 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.859383    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586sj\" (UniqueName: \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.859649 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.859463    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.859649 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.859508    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.860243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.860218    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.861984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.861962    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.920980 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.920902    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586sj\" (UniqueName: \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\") pod \"maas-api-revoke-sub-0165d7d2-v8j4t\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:52.955526 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:52.955486    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:53.341536 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:53.341504    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"]
Oct 31 00:53:53.344062 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:53:53.343995    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898f2564_27c9_440e_b69c_5879d12807a2.slice/crio-822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab WatchSource:0}: Error finding container 822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab: Status 404 returned error can't find the container with id 822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab
Oct 31 00:53:53.391653 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:53.391614    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t" event={"ID":"898f2564-27c9-440e-b69c-5879d12807a2","Type":"ContainerStarted","Data":"822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab"}
Oct 31 00:53:54.398729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:54.398697    6033 generic.go:356] "Generic (PLEG): container finished" podID="898f2564-27c9-440e-b69c-5879d12807a2" containerID="bb6d761bc0c838ad14216de190038a93bedee12511e39ac37fe2f5230e4c339b" exitCode=0
Oct 31 00:53:54.399203 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:54.398783    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t" event={"ID":"898f2564-27c9-440e-b69c-5879d12807a2","Type":"ContainerDied","Data":"bb6d761bc0c838ad14216de190038a93bedee12511e39ac37fe2f5230e4c339b"}
Oct 31 00:53:55.532391 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.532361    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:53:55.687799 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.687700    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\") pod \"898f2564-27c9-440e-b69c-5879d12807a2\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") "
Oct 31 00:53:55.687985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.687875    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\" (UniqueName: \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\") pod \"898f2564-27c9-440e-b69c-5879d12807a2\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") "
Oct 31 00:53:55.687985 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.687899    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\") pod \"898f2564-27c9-440e-b69c-5879d12807a2\" (UID: \"898f2564-27c9-440e-b69c-5879d12807a2\") "
Oct 31 00:53:55.688344 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.688320    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca" pod "898f2564-27c9-440e-b69c-5879d12807a2" (UID: "898f2564-27c9-440e-b69c-5879d12807a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:55.689984 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.689957    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca" pod "898f2564-27c9-440e-b69c-5879d12807a2" (UID: "898f2564-27c9-440e-b69c-5879d12807a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:55.690130 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.690110    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj" pod "898f2564-27c9-440e-b69c-5879d12807a2" (UID: "898f2564-27c9-440e-b69c-5879d12807a2"). InnerVolumeSpecName "kube-api-access-586sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:55.789091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.789008    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586sj\" (UniqueName: \"kubernetes.io/projected/898f2564-27c9-440e-b69c-5879d12807a2-kube-api-access-586sj\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:55.789091 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.789090    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:55.789316 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:55.789106    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898f2564-27c9-440e-b69c-5879d12807a2-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:56.410644 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:56.410607    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t" event={"ID":"898f2564-27c9-440e-b69c-5879d12807a2","Type":"ContainerDied","Data":"822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab"}
Oct 31 00:53:56.410860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:56.410668    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="822d0825dfbd8e492cbfbb7d3cb635a69322d9556338d4b7c7bd8b63fe3b8aab"
Oct 31 00:53:56.410860 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:53:56.410733    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v8j4t"
Oct 31 00:54:04.508824 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.508784    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"]
Oct 31 00:54:04.509446 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.509423    6033 cpu_manager.go:414] "containerMap: removing container" podUID="898f2564-27c9-440e-b69c-5879d12807a2" containerName="revoke-keys"
Oct 31 00:54:04.509541 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.509448    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898f2564-27c9-440e-b69c-5879d12807a2" containerName="revoke-keys"
Oct 31 00:54:04.509613 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.509598    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="898f2564-27c9-440e-b69c-5879d12807a2" containerName="revoke-keys"
Oct 31 00:54:04.512729 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.512700    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.562864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.562823    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkrrm\""
Oct 31 00:54:04.562864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.562840    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:54:04.562864 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.562855    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:54:04.563158 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.562884    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:54:04.581223 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.581186    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.581434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.581329    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.581434 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.581360    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z286g\" (UniqueName: \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.611205 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.611163    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"]
Oct 31 00:54:04.682428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.682381    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.682428 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.682428    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z286g\" (UniqueName: \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.682699 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.682499    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.683147 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.683124    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.684951 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.684928    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.740689 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.740644    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z286g\" (UniqueName: \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\") pod \"maas-api-revoke-sub-6b19338f-9q754\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:04.830841 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:04.830740    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:05.210001 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:05.209913    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"]
Oct 31 00:54:05.214610 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:54:05.214578    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8825e22e_10c9_4c02_99ed_b02de6b4d4a6.slice/crio-2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887 WatchSource:0}: Error finding container 2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887: Status 404 returned error can't find the container with id 2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887
Oct 31 00:54:05.455857 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:05.455817    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754" event={"ID":"8825e22e-10c9-4c02-99ed-b02de6b4d4a6","Type":"ContainerStarted","Data":"2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887"}
Oct 31 00:54:06.462742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:06.462645    6033 generic.go:356] "Generic (PLEG): container finished" podID="8825e22e-10c9-4c02-99ed-b02de6b4d4a6" containerID="501e2de979cb0d97d5453e218898691f663eda4e510691337db7dcca4dbfd908" exitCode=0
Oct 31 00:54:06.462742 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:06.462703    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754" event={"ID":"8825e22e-10c9-4c02-99ed-b02de6b4d4a6","Type":"ContainerDied","Data":"501e2de979cb0d97d5453e218898691f663eda4e510691337db7dcca4dbfd908"}
Oct 31 00:54:07.610988 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.610955    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:07.710634 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.710591    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\") pod \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") "
Oct 31 00:54:07.710800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.710646    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\" (UniqueName: \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\") pod \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") "
Oct 31 00:54:07.710800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.710676    6033 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\") pod \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\" (UID: \"8825e22e-10c9-4c02-99ed-b02de6b4d4a6\") "
Oct 31 00:54:07.711163 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.711138    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca" pod "8825e22e-10c9-4c02-99ed-b02de6b4d4a6" (UID: "8825e22e-10c9-4c02-99ed-b02de6b4d4a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:07.712833 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.712807    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca" pod "8825e22e-10c9-4c02-99ed-b02de6b4d4a6" (UID: "8825e22e-10c9-4c02-99ed-b02de6b4d4a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:54:07.712943 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.712855    6033 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g" pod "8825e22e-10c9-4c02-99ed-b02de6b4d4a6" (UID: "8825e22e-10c9-4c02-99ed-b02de6b4d4a6"). InnerVolumeSpecName "kube-api-access-z286g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:07.811534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.811486    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-xks-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:07.811534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.811526    6033 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z286g\" (UniqueName: \"kubernetes.io/projected/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-kube-api-access-z286g\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:07.811534 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:07.811537    6033 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8825e22e-10c9-4c02-99ed-b02de6b4d4a6-maas-api-openshift-ca\") on node \"ip-10-0-139-66.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:54:08.473762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:08.473728    6033 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754"
Oct 31 00:54:08.473762 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:08.473760    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9q754" event={"ID":"8825e22e-10c9-4c02-99ed-b02de6b4d4a6","Type":"ContainerDied","Data":"2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887"}
Oct 31 00:54:08.474069 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:08.473809    6033 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff04121f66c2a9daf440b79e294829c21a9c54b6528fba091882593c7ea6887"
Oct 31 00:54:16.970363 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:54:16.970324    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:54:17.489243 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:17.489202    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4n4cm"]
Oct 31 00:54:29.550786 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:29.550754    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-szf2t_affbc01f-9f16-4c16-89f4-99edc24e8680/manager/0.log"
Oct 31 00:54:30.177966 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:30.177925    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-z482g_063afbf8-66c3-434a-8e2e-7c9493041154/manager/0.log"
Oct 31 00:54:31.167608 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:31.167577    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6cdml_d67e907e-94d3-4135-8e91-418bcb790b6e/server/0.log"
Oct 31 00:54:31.404438 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:31.404390    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-v8nkx_81f86f98-23df-4c30-ab28-a7b809c2801a/manager/0.log"
Oct 31 00:54:31.672472 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:31.672433    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fgsdn_159c39d8-4d1c-4674-af0d-4d134caaad83/proxy/0.log"
Oct 31 00:54:33.035094 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:33.035065    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v8j4t_898f2564-27c9-440e-b69c-5879d12807a2/revoke-keys/0.log"
Oct 31 00:54:33.259911 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:33.259876    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9q754_8825e22e-10c9-4c02-99ed-b02de6b4d4a6/revoke-keys/0.log"
Oct 31 00:54:33.506865 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:33.506839    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mcncs_5b5c5ef7-644d-4e3b-9cc3-050399c14768/revoke-keys/0.log"
Oct 31 00:54:36.350154 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:36.350084    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8k6m8_442fa4e9-64b3-48ce-ac9f-3bec3e289328/kuadrant-console-plugin/0.log"
Oct 31 00:54:36.976737 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:36.976702    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4n4cm_34e7bbb9-29be-4332-8ddd-34b2e683bd28/limitador/0.log"
Oct 31 00:54:38.805905 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:38.805873    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-r7gvf_ac2b7908-97d9-4867-adef-411bfd62188a/kube-auth-proxy/0.log"
Oct 31 00:54:39.610317 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:39.610260    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-4trk8_aba21fe6-a5e5-49c8-990b-7b3626e8b78f/istio-proxy/0.log"
Oct 31 00:54:40.803583 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:40.803547    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-pqml2_a4a3244b-0d13-42ad-8857-c3d8b83442f4/istio-proxy/0.log"
Oct 31 00:54:41.638150 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:41.638114    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-znmqk_26e036ea-2308-4ba1-bd0f-4955e4bc2f52/istio-proxy/0.log"
Oct 31 00:54:46.970264 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:54:46.970220    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:54:56.976581 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:56.976544    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-szf2t_affbc01f-9f16-4c16-89f4-99edc24e8680/manager/0.log"
Oct 31 00:54:57.599591 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:57.599558    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-z482g_063afbf8-66c3-434a-8e2e-7c9493041154/manager/0.log"
Oct 31 00:54:58.207630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:58.207597    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-6cdml_d67e907e-94d3-4135-8e91-418bcb790b6e/server/0.log"
Oct 31 00:54:58.436122 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:58.436081    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-v8nkx_81f86f98-23df-4c30-ab28-a7b809c2801a/manager/0.log"
Oct 31 00:54:58.716464 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:58.716432    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fgsdn_159c39d8-4d1c-4674-af0d-4d134caaad83/proxy/0.log"
Oct 31 00:54:59.848387 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:54:59.848358    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v8j4t_898f2564-27c9-440e-b69c-5879d12807a2/revoke-keys/0.log"
Oct 31 00:55:00.100190 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:00.100102    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9q754_8825e22e-10c9-4c02-99ed-b02de6b4d4a6/revoke-keys/0.log"
Oct 31 00:55:00.344439 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:00.344407    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-mcncs_5b5c5ef7-644d-4e3b-9cc3-050399c14768/revoke-keys/0.log"
Oct 31 00:55:02.897517 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:02.897470    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8k6m8_442fa4e9-64b3-48ce-ac9f-3bec3e289328/kuadrant-console-plugin/0.log"
Oct 31 00:55:03.434868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:03.434831    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4n4cm_34e7bbb9-29be-4332-8ddd-34b2e683bd28/limitador/0.log"
Oct 31 00:55:05.045211 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:05.045177    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-r7gvf_ac2b7908-97d9-4867-adef-411bfd62188a/kube-auth-proxy/0.log"
Oct 31 00:55:05.894585 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:05.894543    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-4trk8_aba21fe6-a5e5-49c8-990b-7b3626e8b78f/istio-proxy/0.log"
Oct 31 00:55:07.062338 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:07.062309    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-pqml2_a4a3244b-0d13-42ad-8857-c3d8b83442f4/istio-proxy/0.log"
Oct 31 00:55:07.853010 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:55:07.852970    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8f6f87f59-znmqk_26e036ea-2308-4ba1-bd0f-4955e4bc2f52/istio-proxy/0.log"
Oct 31 00:55:16.968925 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:55:16.968883    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:55:46.970086 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:55:46.970047    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:56:16.970792 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:56:16.970703    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:56:30.305759 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:30.305716    6033 scope.go:122] "RemoveContainer" containerID="9407304e8287dd7f25e149cf916781d004c7a33ab2f47799583759a0ccebbcfd"
Oct 31 00:56:36.962843 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:36.962807    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fjfcx_681ea7db-6b38-46c2-91d2-80e27ef3c1af/global-pull-secret-syncer/0.log"
Oct 31 00:56:40.557225 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:40.557184    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w2k95_790c4c63-efba-46c4-b27f-db092797bca3/konnectivity-agent/0.log"
Oct 31 00:56:41.369599 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:41.369557    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-66.us-west-2.compute.internal_4e63feea4e10327df2e22d81d9a9cb4a/haproxy/0.log"
Oct 31 00:56:45.687383 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.687344    6033 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"]
Oct 31 00:56:45.687827 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.687813    6033 cpu_manager.go:414] "containerMap: removing container" podUID="8825e22e-10c9-4c02-99ed-b02de6b4d4a6" containerName="revoke-keys"
Oct 31 00:56:45.687924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.687830    6033 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8825e22e-10c9-4c02-99ed-b02de6b4d4a6" containerName="revoke-keys"
Oct 31 00:56:45.687924 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.687901    6033 memory_manager.go:365] "RemoveStaleState removing state" podUID="8825e22e-10c9-4c02-99ed-b02de6b4d4a6" containerName="revoke-keys"
Oct 31 00:56:45.691215 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.691190    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.827220 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.827180    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8xv\" (UniqueName: \"kubernetes.io/projected/3ea70b38-f142-48d1-8a83-bd7619054bf3-kube-api-access-lj8xv\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.827410 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.827317    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-sys\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.827410 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.827392    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-lib-modules\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.827501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.827438    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-proc\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.827501 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.827480    6033 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-podres\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.833363 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.833334    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"kube-root-ca.crt\""
Oct 31 00:56:45.833497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.833380    6033 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5nxv\"/\"default-dockercfg-k229d\""
Oct 31 00:56:45.833497 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.833432    6033 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5nxv\"/\"openshift-service-ca.crt\""
Oct 31 00:56:45.834763 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.834743    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"]
Oct 31 00:56:45.929062 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.928983    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8xv\" (UniqueName: \"kubernetes.io/projected/3ea70b38-f142-48d1-8a83-bd7619054bf3-kube-api-access-lj8xv\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929110    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-sys\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929140    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-lib-modules\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929171    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-proc\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929213    6033 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-podres\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929262 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929243    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-sys\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929275    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-lib-modules\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929314    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-podres\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.929469 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.929328    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ea70b38-f142-48d1-8a83-bd7619054bf3-proc\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:45.992112 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:45.992002    6033 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8xv\" (UniqueName: \"kubernetes.io/projected/3ea70b38-f142-48d1-8a83-bd7619054bf3-kube-api-access-lj8xv\") pod \"perf-node-gather-daemonset-4sn64\" (UID: \"3ea70b38-f142-48d1-8a83-bd7619054bf3\") " pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:46.004856 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:46.004821    6033 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:46.370868 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:46.370843    6033 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"]
Oct 31 00:56:46.373943 ip-10-0-139-66 kubenswrapper[6033]: W1031 00:56:46.373892    6033 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ea70b38_f142_48d1_8a83_bd7619054bf3.slice/crio-05bc09c3efd7ad9bb848c2eded66f09299021d5fe9f7238fd1743caf637e3aae WatchSource:0}: Error finding container 05bc09c3efd7ad9bb848c2eded66f09299021d5fe9f7238fd1743caf637e3aae: Status 404 returned error can't find the container with id 05bc09c3efd7ad9bb848c2eded66f09299021d5fe9f7238fd1743caf637e3aae
Oct 31 00:56:46.969907 ip-10-0-139-66 kubenswrapper[6033]: E1031 00:56:46.969871    6033 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9lzs" containerName="extractor"
Oct 31 00:56:47.109586 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.109555    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-8k6m8_442fa4e9-64b3-48ce-ac9f-3bec3e289328/kuadrant-console-plugin/0.log"
Oct 31 00:56:47.294620 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.294581    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64" event={"ID":"3ea70b38-f142-48d1-8a83-bd7619054bf3","Type":"ContainerStarted","Data":"45dd194b216ec6278d3dfb16a04bff66226440571f666ecb98d24be439b5f2e4"}
Oct 31 00:56:47.294819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.294638    6033 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64" event={"ID":"3ea70b38-f142-48d1-8a83-bd7619054bf3","Type":"ContainerStarted","Data":"05bc09c3efd7ad9bb848c2eded66f09299021d5fe9f7238fd1743caf637e3aae"}
Oct 31 00:56:47.294819 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.294660    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:47.403662 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.403608    6033 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64" podStartSLOduration=2.403593984 podStartE2EDuration="2.403593984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:47.402727316 +0000 UTC m=+4459.749711214" watchObservedRunningTime="2026-10-31 00:56:47.403593984 +0000 UTC m=+4459.750577881"
Oct 31 00:56:47.921855 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:47.921823    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4n4cm_34e7bbb9-29be-4332-8ddd-34b2e683bd28/limitador/0.log"
Oct 31 00:56:50.808235 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:50.808202    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/alertmanager/0.log"
Oct 31 00:56:51.203760 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:51.203675    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/config-reloader/0.log"
Oct 31 00:56:51.599921 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:51.599888    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/kube-rbac-proxy-web/0.log"
Oct 31 00:56:51.995859 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:51.995780    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/kube-rbac-proxy/0.log"
Oct 31 00:56:52.392641 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:52.392612    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/kube-rbac-proxy-metric/0.log"
Oct 31 00:56:52.787445 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:52.787416    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/prom-label-proxy/0.log"
Oct 31 00:56:53.187598 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:53.187510    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bef3cad5-0aa3-4b87-b3ed-74c60843a276/init-config-reloader/0.log"
Oct 31 00:56:53.309628 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:53.309600    6033 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5nxv/perf-node-gather-daemonset-4sn64"
Oct 31 00:56:54.117375 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:54.117335    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8hddr_d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/kube-state-metrics/0.log"
Oct 31 00:56:54.507182 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:54.507148    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8hddr_d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/kube-rbac-proxy-main/0.log"
Oct 31 00:56:54.900904 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:54.900826    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-8hddr_d9cefa5e-9c3d-42fb-ad15-ad4e1b99036b/kube-rbac-proxy-self/0.log"
Oct 31 00:56:58.600901 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:58.600876    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvtvb_9cc0416c-9ced-475c-8022-97cac5f04340/node-exporter/0.log"
Oct 31 00:56:58.992543 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:58.992469    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvtvb_9cc0416c-9ced-475c-8022-97cac5f04340/kube-rbac-proxy/0.log"
Oct 31 00:56:59.392875 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:59.392844    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvtvb_9cc0416c-9ced-475c-8022-97cac5f04340/init-textfile/0.log"
Oct 31 00:56:59.797836 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:56:59.797802    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6tgwv_a0cc93e9-3bb7-4224-9575-d26cf94807c7/kube-rbac-proxy-main/0.log"
Oct 31 00:57:00.203409 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:00.203327    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6tgwv_a0cc93e9-3bb7-4224-9575-d26cf94807c7/kube-rbac-proxy-self/0.log"
Oct 31 00:57:00.597669 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:00.597639    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6tgwv_a0cc93e9-3bb7-4224-9575-d26cf94807c7/openshift-state-metrics/0.log"
Oct 31 00:57:01.008119 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:01.008093    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/prometheus/0.log"
Oct 31 00:57:01.406800 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:01.406720    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/config-reloader/0.log"
Oct 31 00:57:01.806398 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:01.806365    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/thanos-sidecar/0.log"
Oct 31 00:57:02.201787 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:02.201672    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/kube-rbac-proxy-web/0.log"
Oct 31 00:57:02.598233 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:02.598198    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/kube-rbac-proxy/0.log"
Oct 31 00:57:02.996410 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:02.996331    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/kube-rbac-proxy-thanos/0.log"
Oct 31 00:57:03.391752 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:03.391707    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_47de0f8d-fff4-4d27-bc35-bc926d39e08d/init-config-reloader/0.log"
Oct 31 00:57:03.792516 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:03.792479    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-847hn_91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/prometheus-operator/0.log"
Oct 31 00:57:04.183074 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:04.182975    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-847hn_91c3a0a8-b223-49c9-ad4f-0386b8ff0a9e/kube-rbac-proxy/0.log"
Oct 31 00:57:04.987157 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:04.987126    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54885d6898-mpk4r_389e6e7e-8415-4d41-8583-3bae15af7e96/telemeter-client/0.log"
Oct 31 00:57:05.386670 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:05.386639    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54885d6898-mpk4r_389e6e7e-8415-4d41-8583-3bae15af7e96/reload/0.log"
Oct 31 00:57:05.781677 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:05.781637    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54885d6898-mpk4r_389e6e7e-8415-4d41-8583-3bae15af7e96/kube-rbac-proxy/0.log"
Oct 31 00:57:06.186365 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:06.186285    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/thanos-query/0.log"
Oct 31 00:57:06.581688 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:06.581645    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/kube-rbac-proxy-web/0.log"
Oct 31 00:57:06.977198 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:06.977119    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/kube-rbac-proxy/0.log"
Oct 31 00:57:07.372789 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:07.372749    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/prom-label-proxy/0.log"
Oct 31 00:57:07.766315 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:07.766285    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/kube-rbac-proxy-rules/0.log"
Oct 31 00:57:08.159630 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:08.159551    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86dfbdff78-j4gh6_7ed2164b-7445-41c3-a449-bc2b0168ba72/kube-rbac-proxy-metrics/0.log"
Oct 31 00:57:09.894350 ip-10-0-139-66 kubenswrapper[6033]: I1031 00:57:09.894321    6033 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546bf7cb95-r7gvf_ac2b7908-97d9-4867-adef-411bfd62188a/kube-auth-proxy/0.log"